+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit
+ bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20190507/A1c --output-run DEEP/20190507/A1c/science#step1/20240309T205315Z --qgraph pipeline.qgraph
lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process.
lsst.ctrl.bps.drivers INFO: Starting submission process
lsst.ctrl.bps.drivers INFO: Initializing execution environment
lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 6.0930 seconds; current memory usage: 0.184 Gibyte, delta: 0.044 Gibyte, peak delta: 0.055 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.196 Gibyte (main), 0.000 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph)
lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0110 seconds
lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0241 seconds
lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/pipeline_orig.qgraph DEEP/20190507/A1c/science#step1/20240309T205315Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/pipeline.qgraph 
lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 8.1672 seconds
lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 23.6183 seconds; current memory usage: 0.385 Gibyte, delta: 0.201 Gibyte, peak delta: 0.190 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.385 Gibyte (main), 0.369 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs)
lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0435 seconds; current memory usage: 0.385 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.385 Gibyte (main), 0.369 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 1116 cluster(s)
lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow)
lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20190507_A1c_science#step1_20240309T205315Z'
lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.3176 seconds; current memory usage: 0.385 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.385 Gibyte (main), 0.369 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 1118 job(s) (including final)
lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow)
parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses
    s_addresses.add(address_by_interface(interface))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper
    retval = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface
    return socket.inet_ntoa(fcntl.ioctl(
                            ^^^^^^^^^^^^
OSError: [Errno 99] Cannot assign requested address
lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z
lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.7623 seconds; current memory usage: 0.387 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.387 Gibyte (main), 0.369 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting submit stage
lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution
parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000
parsl.dataflow.dflow INFO: Starting DataFlowKernel with config
Config(
    app_cache=True, 
    checkpoint_files=None, 
    checkpoint_mode='task_exit', 
    checkpoint_period=None, 
    executors=[MultiHighThroughputExecutor()], 
    garbage_collect=True, 
    initialize_logging=True, 
    internal_tasks_max_threads=10, 
    max_idletime=120.0, 
    monitoring=None, 
    retries=1, 
    retry_handler=None, 
    run_dir='runinfo', 
    strategy='simple', 
    usage_tracking=False
)
parsl.dataflow.dflow INFO: Parsl version: 2023.06.12
parsl.usage_tracking.usage DEBUG: Tracking status: False
parsl.dataflow.dflow INFO: Run id is: d208d530-3256-498c-817b-669803b13170
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py
parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name
parsl.dataflow.memoization INFO: App caching initialized
parsl.dataflow.strategy DEBUG: Scaling strategy: simple
parsl.executors.high_throughput.executor DEBUG: Starting queue management thread
parsl.executors.high_throughput.executor DEBUG: queue management worker starting
parsl.executors.high_throughput.executor DEBUG: Started queue management thread
Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z
parsl.executors.high_throughput.executor DEBUG: Created management thread: <Thread(HTEX-Queue-Management-Thread, started daemon 22720741742336)>
parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id={block_id} --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider:
<proc_lsst.multi.MultiProvider object at 0x14aa17148f90>
parsl.executors.status_handling INFO: Scaling out by 1 blocks
parsl.executors.status_handling INFO: Allocated block ID 0
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0
2024-03-09 20:54:26 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-0
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-0
2024-03-09 20:54:26 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:26 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-09 20:54:26 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:26 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2 to local
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2 to local
parsl.providers.local.local DEBUG: Launching in remote mode
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 0 executors
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 20:54:26 proc_lsst.multi:170 [INFO]  [multi] job_id 51896
proc_lsst.multi INFO: [multi] job_id 51896
2024-03-09 20:54:26 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:26 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:26 proc_lsst.multi:178 [INFO]  [multi] provider local accepted submit and returned 51896
proc_lsst.multi INFO: [multi] provider local accepted submit and returned 51896
parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 51896
parsl.dataflow.job_status_poller DEBUG: Adding executor multi
parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 0 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: <AppFuture at 0x14aa19d3b010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 0 has memoization hash c71f2ec4bc90d67631ec679fc947ae05
parsl.dataflow.memoization INFO: Task 0 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fc220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1
parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/bc39d58e-287f-4f59-8466-6cb8ccc18d09_isr_855822_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/bc39d58e-287f-4f59-8466-6cb8ccc18d09_isr_855822_22.stderr
parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1 submitted for App characterizeImage, waiting on task 0
parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: <AppFuture at 0x14aa19b39810 state=pending>
parsl.dataflow.dflow DEBUG: Task 1 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 2 submitted for App calibrate, waiting on task 1
parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: <AppFuture at 0x14aa1832f750 state=pending>
parsl.dataflow.dflow DEBUG: Task 2 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 3 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: <AppFuture at 0x14aa18159d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 9c7cd10186f7f718f805aa968a232687
parsl.dataflow.memoization INFO: Task 3 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fcae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 2
parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/2ca5dd9a-a895-47e7-b6af-063c90df40d0_isr_855911_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/2ca5dd9a-a895-47e7-b6af-063c90df40d0_isr_855911_33.stderr
parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 4 submitted for App characterizeImage, waiting on task 3
parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: <AppFuture at 0x14aa17667250 state=pending>
parsl.dataflow.dflow DEBUG: Task 4 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 5 submitted for App calibrate, waiting on task 4
parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: <AppFuture at 0x14aa17e3ee90 state=pending>
parsl.dataflow.dflow DEBUG: Task 5 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 6 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: <AppFuture at 0x14aa18e15f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 6 has memoization hash d41eaa319e0976cbd403945f6ef1ac82
parsl.dataflow.memoization INFO: Task 6 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fcd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 3
parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/0e6c775b-f198-4540-b7ca-6afc65c32e13_isr_855822_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/0e6c775b-f198-4540-b7ca-6afc65c32e13_isr_855822_13.stderr
parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 7 submitted for App characterizeImage, waiting on task 6
parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: <AppFuture at 0x14aa17589c10 state=pending>
parsl.dataflow.dflow DEBUG: Task 7 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 8 submitted for App calibrate, waiting on task 7
parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: <AppFuture at 0x14aa17a81fd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 8 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 9 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: <AppFuture at 0x14aa1878e890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 9 has memoization hash ac4efccc3709ebb02c44e7b6e79a19e3
parsl.dataflow.memoization INFO: Task 9 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fcfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 4
parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/d305c9b4-bcf1-4083-b5eb-bbcb20eb33d3_isr_855820_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/d305c9b4-bcf1-4083-b5eb-bbcb20eb33d3_isr_855820_32.stderr
parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 10 submitted for App characterizeImage, waiting on task 9
parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: <AppFuture at 0x14aa27b6cc50 state=pending>
parsl.dataflow.dflow DEBUG: Task 10 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 11 submitted for App calibrate, waiting on task 10
parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: <AppFuture at 0x14aa18a924d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 11 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 12 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: <AppFuture at 0x14aa18a90950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 12 has memoization hash bc0c11569bb434b1197674500359d246
parsl.dataflow.memoization INFO: Task 12 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fd580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 5
parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/9f963235-ea2e-4b32-b719-0d08d19abe6a_isr_855911_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/9f963235-ea2e-4b32-b719-0d08d19abe6a_isr_855911_57.stderr
parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 13 submitted for App characterizeImage, waiting on task 12
parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: <AppFuture at 0x14aa18a92e10 state=pending>
parsl.dataflow.dflow DEBUG: Task 13 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 14 submitted for App calibrate, waiting on task 13
parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: <AppFuture at 0x14aa1729b410 state=pending>
parsl.dataflow.dflow DEBUG: Task 14 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 15 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: <AppFuture at 0x14aa17d19410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 15 has memoization hash b2e47e9b77b6a330505ac0af31e84c92
parsl.dataflow.memoization INFO: Task 15 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 6
parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/fa913612-5e28-4c5f-9c8f-e8719d5c54ed_isr_855909_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/fa913612-5e28-4c5f-9c8f-e8719d5c54ed_isr_855909_45.stderr
parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 16 submitted for App characterizeImage, waiting on task 15
parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: <AppFuture at 0x14aa18f6c5d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 16 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 17 submitted for App calibrate, waiting on task 16
parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: <AppFuture at 0x14aa17d1a590 state=pending>
parsl.dataflow.dflow DEBUG: Task 17 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 18 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: <AppFuture at 0x14aa18f6e210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 80770de630fe4d2f4fbc5e9e36a58e13
parsl.dataflow.memoization INFO: Task 18 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fdbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 7
parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/c61aa638-084e-4483-91ab-8fbcdc3923e1_isr_855822_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/c61aa638-084e-4483-91ab-8fbcdc3923e1_isr_855822_10.stderr
parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 19 submitted for App characterizeImage, waiting on task 18
parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: <AppFuture at 0x14aa18f6f550 state=pending>
parsl.dataflow.dflow DEBUG: Task 19 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 20 submitted for App calibrate, waiting on task 19
parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: <AppFuture at 0x14aa19b16550 state=pending>
parsl.dataflow.dflow DEBUG: Task 20 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 21 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: <AppFuture at 0x14aa19b17350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 0e3165c7e18be337269c97761a623f7a
parsl.dataflow.memoization INFO: Task 21 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fdee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 8
parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/6e0135bb-233d-40e6-b768-b0e91a61e8df_isr_855909_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/6e0135bb-233d-40e6-b768-b0e91a61e8df_isr_855909_4.stderr
parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 22 submitted for App characterizeImage, waiting on task 21
parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: <AppFuture at 0x14aa19b161d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 23 submitted for App calibrate, waiting on task 22
parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: <AppFuture at 0x14aa17b1ae50 state=pending>
parsl.dataflow.dflow DEBUG: Task 23 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 24 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: <AppFuture at 0x14aa18cabfd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 9f43ee98fa484ca34a545342548830d0
parsl.dataflow.memoization INFO: Task 24 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169ff060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 9
parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/25977ed8-d795-46e2-8f46-894a614d9923_isr_855822_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/25977ed8-d795-46e2-8f46-894a614d9923_isr_855822_9.stderr
parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 25 submitted for App characterizeImage, waiting on task 24
parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: <AppFuture at 0x14aa17cba550 state=pending>
parsl.dataflow.dflow DEBUG: Task 25 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 26 submitted for App calibrate, waiting on task 25
parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: <AppFuture at 0x14aa185b2850 state=pending>
parsl.dataflow.dflow DEBUG: Task 26 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 27 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: <AppFuture at 0x14aa1775da50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 2183a36d7c59cf4e408f2de6dad43297
parsl.dataflow.memoization INFO: Task 27 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 10
parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/e34aad39-a850-4d6a-82cc-4a92ef347cc3_isr_855910_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/e34aad39-a850-4d6a-82cc-4a92ef347cc3_isr_855910_1.stderr
parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 28 submitted for App characterizeImage, waiting on task 27
parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: <AppFuture at 0x14aa17bb6bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 28 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 29 submitted for App calibrate, waiting on task 28
parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: <AppFuture at 0x14aa1830ddd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 29 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 30 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: <AppFuture at 0x14aa1879cd10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 7348f312bacb63cecb5b879cb5c1971c
parsl.dataflow.memoization INFO: Task 30 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 11
parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/156e09da-e1d4-45e1-acb5-2a19ece72141_isr_855822_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/156e09da-e1d4-45e1-acb5-2a19ece72141_isr_855822_8.stderr
parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 31 submitted for App characterizeImage, waiting on task 30
parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: <AppFuture at 0x14aa1879dc90 state=pending>
parsl.dataflow.dflow DEBUG: Task 31 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 32 submitted for App calibrate, waiting on task 31
parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: <AppFuture at 0x14aa175bf590 state=pending>
parsl.dataflow.dflow DEBUG: Task 32 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 33 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: <AppFuture at 0x14aa174c1410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 7eb106b1da7db62332917c20eec76b4e
parsl.dataflow.memoization INFO: Task 33 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fe700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 12
parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/4b845abe-c313-4d31-b447-e0151279eeb5_isr_855910_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/4b845abe-c313-4d31-b447-e0151279eeb5_isr_855910_56.stderr
parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 34 submitted for App characterizeImage, waiting on task 33
parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: <AppFuture at 0x14aa176995d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 34 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 35 submitted for App calibrate, waiting on task 34
parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: <AppFuture at 0x14aa18ca0050 state=pending>
parsl.dataflow.dflow DEBUG: Task 35 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 36 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: <AppFuture at 0x14aa18f76390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 22b712c2bd9ec854036f1310ca03ce70
parsl.dataflow.memoization INFO: Task 36 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fe3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 13
parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/faf088f9-84e9-4189-92b4-169f97e6ebce_isr_855909_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/faf088f9-84e9-4189-92b4-169f97e6ebce_isr_855909_34.stderr
parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 37 submitted for App characterizeImage, waiting on task 36
parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: <AppFuture at 0x14aa18f75350 state=pending>
parsl.dataflow.dflow DEBUG: Task 37 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 38 submitted for App calibrate, waiting on task 37
parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: <AppFuture at 0x14aa173cbfd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 38 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 39 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: <AppFuture at 0x14aa17c77490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 39 has memoization hash d085d2350d706208e3cd79a536db4f78
parsl.dataflow.memoization INFO: Task 39 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fe020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 14
parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/3732fe83-9831-4d50-9663-71cf5b0455cc_isr_855909_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/3732fe83-9831-4d50-9663-71cf5b0455cc_isr_855909_31.stderr
parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 40 submitted for App characterizeImage, waiting on task 39
parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: <AppFuture at 0x14aa18dcae50 state=pending>
parsl.dataflow.dflow DEBUG: Task 40 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 41 submitted for App calibrate, waiting on task 40
parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: <AppFuture at 0x14aa18dcad90 state=pending>
parsl.dataflow.dflow DEBUG: Task 41 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 42 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: <AppFuture at 0x14aa1b6f8390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 4934caed02d9ba35e683642c8da84bcc
parsl.dataflow.memoization INFO: Task 42 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169ff560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 15
parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/a2bfdb1d-931c-4f98-9961-abfb57a4737b_isr_855820_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/a2bfdb1d-931c-4f98-9961-abfb57a4737b_isr_855820_58.stderr
parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 43 submitted for App characterizeImage, waiting on task 42
parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: <AppFuture at 0x14aa19b14610 state=pending>
parsl.dataflow.dflow DEBUG: Task 43 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 44 submitted for App calibrate, waiting on task 43
parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: <AppFuture at 0x14aa19dc4e90 state=pending>
parsl.dataflow.dflow DEBUG: Task 44 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 45 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: <AppFuture at 0x14aa19dc7390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 2a38537c09a964e6c2831af6714ad131
parsl.dataflow.memoization INFO: Task 45 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169ff880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 16
parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/ced7aad1-1d5c-4875-bd34-f81a0372b49c_isr_855820_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/ced7aad1-1d5c-4875-bd34-f81a0372b49c_isr_855820_27.stderr
parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 46 submitted for App characterizeImage, waiting on task 45
parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: <AppFuture at 0x14aa19dc58d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 46 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 47 submitted for App calibrate, waiting on task 46
parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: <AppFuture at 0x14aa18e35b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 47 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 48 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: <AppFuture at 0x14aa18b21650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 744eb2631fb9fa64539d15a454d622ca
parsl.dataflow.memoization INFO: Task 48 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169ffba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 17
parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/8e01a36d-2979-4279-bb83-9fc0f1867c8e_isr_855909_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/8e01a36d-2979-4279-bb83-9fc0f1867c8e_isr_855909_39.stderr
parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 49 submitted for App characterizeImage, waiting on task 48
parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: <AppFuture at 0x14aa18b23090 state=pending>
parsl.dataflow.dflow DEBUG: Task 49 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 50 submitted for App calibrate, waiting on task 49
parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: <AppFuture at 0x14aa18e4d210 state=pending>
parsl.dataflow.dflow DEBUG: Task 50 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 51 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: <AppFuture at 0x14aa18e4fb50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 60d178f14c015bbbf25164278e20b920
parsl.dataflow.memoization INFO: Task 51 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169ffec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 18
parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/218562ee-24ca-4966-99fd-3265df808d47_isr_855822_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/218562ee-24ca-4966-99fd-3265df808d47_isr_855822_48.stderr
parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 52 submitted for App characterizeImage, waiting on task 51
parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: <AppFuture at 0x14aa17544450 state=pending>
parsl.dataflow.dflow DEBUG: Task 52 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 53 submitted for App calibrate, waiting on task 52
parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: <AppFuture at 0x14aa1767ac90 state=pending>
parsl.dataflow.dflow DEBUG: Task 53 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 54 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: <AppFuture at 0x14aa19dd5dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 5806a75423fe6c7f67dc74d3df2c78ae
parsl.dataflow.memoization INFO: Task 54 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 19
parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/b0495512-328b-4c97-8df6-f9486db34b9e_isr_855911_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/b0495512-328b-4c97-8df6-f9486db34b9e_isr_855911_22.stderr
parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 55 submitted for App characterizeImage, waiting on task 54
parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: <AppFuture at 0x14aa19dd64d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 55 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 56 submitted for App calibrate, waiting on task 55
parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: <AppFuture at 0x14aa19dd4e50 state=pending>
parsl.dataflow.dflow DEBUG: Task 56 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 57 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: <AppFuture at 0x14aa19dd5c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 211830d6fe1a3251251b25d48b0ddec1
parsl.dataflow.memoization INFO: Task 57 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 20
parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/da116374-d7ef-4458-914d-0218b259358e_isr_855822_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/da116374-d7ef-4458-914d-0218b259358e_isr_855822_41.stderr
parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 58 submitted for App characterizeImage, waiting on task 57
parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: <AppFuture at 0x14aa19dd7990 state=pending>
parsl.dataflow.dflow DEBUG: Task 58 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 59 submitted for App calibrate, waiting on task 58
parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: <AppFuture at 0x14aa17279850 state=pending>
parsl.dataflow.dflow DEBUG: Task 59 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 60 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: <AppFuture at 0x14aa180d9b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 60 has memoization hash b7fa37d7e40f4e14ad4a246b6f43a394
parsl.dataflow.memoization INFO: Task 60 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 21
parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/35bd3de8-b48a-4883-bddd-96bf771c4460_isr_855910_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/35bd3de8-b48a-4883-bddd-96bf771c4460_isr_855910_20.stderr
parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 61 submitted for App characterizeImage, waiting on task 60
parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: <AppFuture at 0x14aa18941e90 state=pending>
parsl.dataflow.dflow DEBUG: Task 61 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 62 submitted for App calibrate, waiting on task 61
parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: <AppFuture at 0x14aa185c2090 state=pending>
parsl.dataflow.dflow DEBUG: Task 62 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 63 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: <AppFuture at 0x14aa185c3f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 2771011aa5692236f5b92c631eeab51b
parsl.dataflow.memoization INFO: Task 63 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 22
parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/a3aae6ae-afac-4129-82f6-409b9c9f6719_isr_855822_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/a3aae6ae-afac-4129-82f6-409b9c9f6719_isr_855822_11.stderr
parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 64 submitted for App characterizeImage, waiting on task 63
parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: <AppFuture at 0x14aa18478690 state=pending>
parsl.dataflow.dflow DEBUG: Task 64 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 65 submitted for App calibrate, waiting on task 64
parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: <AppFuture at 0x14aa18242f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 65 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 66 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: <AppFuture at 0x14aa181a5390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 66 has memoization hash feb930155cd1ac4465090205fe48e9d6
parsl.dataflow.memoization INFO: Task 66 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 23
parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/e25e2421-b5cd-476a-93cb-1c478625d519_isr_855909_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/e25e2421-b5cd-476a-93cb-1c478625d519_isr_855909_2.stderr
parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 67 submitted for App characterizeImage, waiting on task 66
parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: <AppFuture at 0x14aa1847ab50 state=pending>
parsl.dataflow.dflow DEBUG: Task 67 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 68 submitted for App calibrate, waiting on task 67
parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: <AppFuture at 0x14aa19cfb410 state=pending>
parsl.dataflow.dflow DEBUG: Task 68 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 69 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: <AppFuture at 0x14aa17592d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 09746aa78871951a55bfa3265a23be31
parsl.dataflow.memoization INFO: Task 69 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 24
parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/5b572b29-b459-4dd5-b65a-824f2254a1e3_isr_855909_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/5b572b29-b459-4dd5-b65a-824f2254a1e3_isr_855909_37.stderr
parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 70 submitted for App characterizeImage, waiting on task 69
parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: <AppFuture at 0x14aa17550050 state=pending>
parsl.dataflow.dflow DEBUG: Task 70 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 71 submitted for App calibrate, waiting on task 70
parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: <AppFuture at 0x14aa17551210 state=pending>
parsl.dataflow.dflow DEBUG: Task 71 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 72 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: <AppFuture at 0x14aa19b25490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 72 has memoization hash aee033a5e0a883b2acdad73f9713644d
parsl.dataflow.memoization INFO: Task 72 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 25
parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/6a707045-5237-4f9d-84ee-d5dc47fc1ba4_isr_855910_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/6a707045-5237-4f9d-84ee-d5dc47fc1ba4_isr_855910_45.stderr
parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 73 submitted for App characterizeImage, waiting on task 72
parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: <AppFuture at 0x14aa19b249d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 73 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 74 submitted for App calibrate, waiting on task 73
parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: <AppFuture at 0x14aa19b26090 state=pending>
parsl.dataflow.dflow DEBUG: Task 74 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 75 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: <AppFuture at 0x14aa18de8350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 40285742b27a80ae6e5b2c601c510166
parsl.dataflow.memoization INFO: Task 75 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 26
parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/47943823-dfae-49b6-8a8e-6fbc789442fb_isr_855822_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/47943823-dfae-49b6-8a8e-6fbc789442fb_isr_855822_2.stderr
parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 76 submitted for App characterizeImage, waiting on task 75
parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: <AppFuture at 0x14aa18de9490 state=pending>
parsl.dataflow.dflow DEBUG: Task 76 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 77 submitted for App calibrate, waiting on task 76
parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: <AppFuture at 0x14aa17945a10 state=pending>
parsl.dataflow.dflow DEBUG: Task 77 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 78 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: <AppFuture at 0x14aa179470d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 78 has memoization hash ccb59a432459a7ae58d6335dbb330d16
parsl.dataflow.memoization INFO: Task 78 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 27
parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/b5255453-23d4-4325-b5f6-75509c071f6e_isr_855821_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/b5255453-23d4-4325-b5f6-75509c071f6e_isr_855821_16.stderr
parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 79 submitted for App characterizeImage, waiting on task 78
parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: <AppFuture at 0x14aa18afce90 state=pending>
parsl.dataflow.dflow DEBUG: Task 79 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 80 submitted for App calibrate, waiting on task 79
parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: <AppFuture at 0x14aa27ba7590 state=pending>
parsl.dataflow.dflow DEBUG: Task 80 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 81 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: <AppFuture at 0x14aa27ba61d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 96d36506deda9389e18fc6a892ef9dac
parsl.dataflow.memoization INFO: Task 81 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 28
parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/96ad5f4e-d5cb-4304-8f0f-9dce5ac62085_isr_855820_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/96ad5f4e-d5cb-4304-8f0f-9dce5ac62085_isr_855820_11.stderr
parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 82 submitted for App characterizeImage, waiting on task 81
parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: <AppFuture at 0x14aa18cec690 state=pending>
parsl.dataflow.dflow DEBUG: Task 82 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 83 submitted for App calibrate, waiting on task 82
parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: <AppFuture at 0x14aa1bbf02d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 83 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 84 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: <AppFuture at 0x14aa18d37bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 84 has memoization hash a87423074a1d5bf78b40171ec326329a
parsl.dataflow.memoization INFO: Task 84 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 29
parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/7ee11a91-1eef-40ee-8030-6528934a4944_isr_855821_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/7ee11a91-1eef-40ee-8030-6528934a4944_isr_855821_12.stderr
parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 85 submitted for App characterizeImage, waiting on task 84
parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: <AppFuture at 0x14aa18d37d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 85 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 86 submitted for App calibrate, waiting on task 85
parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: <AppFuture at 0x14aa18d352d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 86 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 87 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: <AppFuture at 0x14aa171d3050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 87 has memoization hash b35afb0315bf991d39f3a41dd9ce12ff
parsl.dataflow.memoization INFO: Task 87 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 30
parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/ed4b9191-ad41-47b6-a2ba-c0bea4275aea_isr_855820_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/ed4b9191-ad41-47b6-a2ba-c0bea4275aea_isr_855820_43.stderr
parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 88 submitted for App characterizeImage, waiting on task 87
parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: <AppFuture at 0x14aa18d34790 state=pending>
parsl.dataflow.dflow DEBUG: Task 88 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 89 submitted for App calibrate, waiting on task 88
parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: <AppFuture at 0x14aa17df5a90 state=pending>
parsl.dataflow.dflow DEBUG: Task 89 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 90 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: <AppFuture at 0x14aa17df73d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 9d8bd4496964db03e555fe85d7d531eb
parsl.dataflow.memoization INFO: Task 90 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 31
parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/0d62d0fc-d715-4efa-885d-1081b0e5810c_isr_855910_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/0d62d0fc-d715-4efa-885d-1081b0e5810c_isr_855910_37.stderr
parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 91 submitted for App characterizeImage, waiting on task 90
parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: <AppFuture at 0x14aa177f3990 state=pending>
parsl.dataflow.dflow DEBUG: Task 91 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 92 submitted for App calibrate, waiting on task 91
parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: <AppFuture at 0x14aa18f51790 state=pending>
parsl.dataflow.dflow DEBUG: Task 92 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 93 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: <AppFuture at 0x14aa18f53d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 353abf2401e115a909a5d82313bf99b2
parsl.dataflow.memoization INFO: Task 93 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 32
parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/334a2251-3dbb-4c74-b6a5-3ce5cd77ad38_isr_855910_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/334a2251-3dbb-4c74-b6a5-3ce5cd77ad38_isr_855910_55.stderr
parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 94 submitted for App characterizeImage, waiting on task 93
parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: <AppFuture at 0x14aa18f75a10 state=pending>
parsl.dataflow.dflow DEBUG: Task 94 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 95 submitted for App calibrate, waiting on task 94
parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: <AppFuture at 0x14aa19df4790 state=pending>
parsl.dataflow.dflow DEBUG: Task 95 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 96 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: <AppFuture at 0x14aa19df48d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 7d0ca43d5bb715512d23af29cc6711ab
parsl.dataflow.memoization INFO: Task 96 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 33
parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/01558511-a086-4ce1-9e0d-bd8fe22f41de_isr_855911_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/01558511-a086-4ce1-9e0d-bd8fe22f41de_isr_855911_55.stderr
parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 97 submitted for App characterizeImage, waiting on task 96
parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: <AppFuture at 0x14aa19df5750 state=pending>
parsl.dataflow.dflow DEBUG: Task 97 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 98 submitted for App calibrate, waiting on task 97
parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: <AppFuture at 0x14aa19de4d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 98 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 99 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: <AppFuture at 0x14aa19de6d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 25c1c892b5ac337e1e9a95a7711f5e7e
parsl.dataflow.memoization INFO: Task 99 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 34
parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/0321f377-6177-4332-9deb-7eb6ea92ff59_isr_855910_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/0321f377-6177-4332-9deb-7eb6ea92ff59_isr_855910_8.stderr
parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 100 submitted for App characterizeImage, waiting on task 99
parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: <AppFuture at 0x14aa19de5a10 state=pending>
parsl.dataflow.dflow DEBUG: Task 100 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 101 submitted for App calibrate, waiting on task 100
parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: <AppFuture at 0x14aa19de6310 state=pending>
parsl.dataflow.dflow DEBUG: Task 101 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 102 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: <AppFuture at 0x14aa19de7f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 235232c620a3adf251b8bce362b8ff80
parsl.dataflow.memoization INFO: Task 102 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 35
parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/805ebeb8-9def-4e6e-8a2c-b24a2f850d6b_isr_855911_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/805ebeb8-9def-4e6e-8a2c-b24a2f850d6b_isr_855911_38.stderr
parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 103 submitted for App characterizeImage, waiting on task 102
parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: <AppFuture at 0x14aa19de70d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 103 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 104 submitted for App calibrate, waiting on task 103
parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: <AppFuture at 0x14aa19b09ad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 104 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 105 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: <AppFuture at 0x14aa19b09410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 3d2f04fe1e4a9a55c43a0223c93a5c8e
parsl.dataflow.memoization INFO: Task 105 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 36
parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/b2053b0c-bd83-41fe-91b6-7896895a98df_isr_855909_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/b2053b0c-bd83-41fe-91b6-7896895a98df_isr_855909_49.stderr
parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 106 submitted for App characterizeImage, waiting on task 105
parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: <AppFuture at 0x14aa19b08e50 state=pending>
parsl.dataflow.dflow DEBUG: Task 106 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 107 submitted for App calibrate, waiting on task 106
parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: <AppFuture at 0x14aa19b09e10 state=pending>
parsl.dataflow.dflow DEBUG: Task 107 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 108 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: <AppFuture at 0x14aa17371b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 84cf7b6b62eec8696032bcd74c7eae02
parsl.dataflow.memoization INFO: Task 108 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 37
parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a573b3b8-ad8b-4057-b434-d69f6b120e1b_isr_855911_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a573b3b8-ad8b-4057-b434-d69f6b120e1b_isr_855911_59.stderr
parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 109 submitted for App characterizeImage, waiting on task 108
parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: <AppFuture at 0x14aa17372210 state=pending>
parsl.dataflow.dflow DEBUG: Task 109 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 110 submitted for App calibrate, waiting on task 109
parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: <AppFuture at 0x14aa19d47590 state=pending>
parsl.dataflow.dflow DEBUG: Task 110 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 111 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: <AppFuture at 0x14aa19d46f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 317d729119b5e6abe10c11fa34710d9f
parsl.dataflow.memoization INFO: Task 111 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 38
parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/0d20626d-1345-4cc4-9752-2360c43eaadb_isr_855909_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/0d20626d-1345-4cc4-9752-2360c43eaadb_isr_855909_51.stderr
parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 112 submitted for App characterizeImage, waiting on task 111
parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: <AppFuture at 0x14aa19d47d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 112 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 113 submitted for App calibrate, waiting on task 112
parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: <AppFuture at 0x14aa19ce8f50 state=pending>
parsl.dataflow.dflow DEBUG: Task 113 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 114 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: <AppFuture at 0x14aa19ceb490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 6b45644cbcd85f8bdac10fb16c1c18cc
parsl.dataflow.memoization INFO: Task 114 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a400e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 39
parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/37420b54-296a-412a-ad54-cebc4f021fa3_isr_855821_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/37420b54-296a-412a-ad54-cebc4f021fa3_isr_855821_32.stderr
parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 115 submitted for App characterizeImage, waiting on task 114
parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: <AppFuture at 0x14aa19ce8690 state=pending>
parsl.dataflow.dflow DEBUG: Task 115 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 116 submitted for App calibrate, waiting on task 115
parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: <AppFuture at 0x14aa19ce8ed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 116 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 117 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: <AppFuture at 0x14aa1777de10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 04864d64154bf38c75f882b77d99ad6b
parsl.dataflow.memoization INFO: Task 117 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a40400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 40
parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/69a4b2ba-a675-4379-8015-19ccadf0f521_isr_855822_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/69a4b2ba-a675-4379-8015-19ccadf0f521_isr_855822_19.stderr
parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 118 submitted for App characterizeImage, waiting on task 117
parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: <AppFuture at 0x14aa1777e890 state=pending>
parsl.dataflow.dflow DEBUG: Task 118 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 119 submitted for App calibrate, waiting on task 118
parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: <AppFuture at 0x14aa19cd8b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 119 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 120 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: <AppFuture at 0x14aa19cdb3d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 0a5eeb2b5c80909f7a5f23f667a4e0bd
parsl.dataflow.memoization INFO: Task 120 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a40720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 41
parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/8a779896-41bc-4ef4-9a47-67ad503b3943_isr_855820_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/8a779896-41bc-4ef4-9a47-67ad503b3943_isr_855820_24.stderr
parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 121 submitted for App characterizeImage, waiting on task 120
parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: <AppFuture at 0x14aa19cda3d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 121 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 122 submitted for App calibrate, waiting on task 121
parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: <AppFuture at 0x14aa19cd9290 state=pending>
parsl.dataflow.dflow DEBUG: Task 122 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 123 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: <AppFuture at 0x14aa19cda7d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 3ea4c1143889660badcdd28a4d41a834
parsl.dataflow.memoization INFO: Task 123 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a40a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 42
parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/2aedf1d3-7761-4c55-8ce7-cc04a950b656_isr_855822_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/2aedf1d3-7761-4c55-8ce7-cc04a950b656_isr_855822_56.stderr
parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 124 submitted for App characterizeImage, waiting on task 123
parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: <AppFuture at 0x14aa19b03e90 state=pending>
parsl.dataflow.dflow DEBUG: Task 124 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 125 submitted for App calibrate, waiting on task 124
parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: <AppFuture at 0x14aa19b02cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 125 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 126 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: <AppFuture at 0x14aa19b02950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 126 has memoization hash c3397154156a59932666e7e61a96105f
parsl.dataflow.memoization INFO: Task 126 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a40d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 43
parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/9232864f-0d9b-4a55-9656-45cf0de8ce48_isr_855909_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/9232864f-0d9b-4a55-9656-45cf0de8ce48_isr_855909_60.stderr
parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 127 submitted for App characterizeImage, waiting on task 126
parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: <AppFuture at 0x14aa19b00d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 127 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 128 submitted for App calibrate, waiting on task 127
parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: <AppFuture at 0x14aa19ccec10 state=pending>
parsl.dataflow.dflow DEBUG: Task 128 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 129 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: <AppFuture at 0x14aa19ccf790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 129 has memoization hash f35ecd19dcb1687552952d854241c293
parsl.dataflow.memoization INFO: Task 129 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a41080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 44
parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/97b22ee0-7f3d-418b-b816-1e4727cc2497_isr_855909_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/97b22ee0-7f3d-418b-b816-1e4727cc2497_isr_855909_59.stderr
parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 130 submitted for App characterizeImage, waiting on task 129
parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: <AppFuture at 0x14aa19ccf590 state=pending>
parsl.dataflow.dflow DEBUG: Task 130 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 131 submitted for App calibrate, waiting on task 130
parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: <AppFuture at 0x14aa18ad0590 state=pending>
parsl.dataflow.dflow DEBUG: Task 131 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 132 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: <AppFuture at 0x14aa18ad3b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 975119d87b02695f31e20febae116d0d
parsl.dataflow.memoization INFO: Task 132 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a413a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 45
parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/b6adf367-2025-4337-8221-49d15cdbb732_isr_855821_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/b6adf367-2025-4337-8221-49d15cdbb732_isr_855821_34.stderr
parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 133 submitted for App characterizeImage, waiting on task 132
parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: <AppFuture at 0x14aa18ad1150 state=pending>
parsl.dataflow.dflow DEBUG: Task 133 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 134 submitted for App calibrate, waiting on task 133
parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: <AppFuture at 0x14aa19931210 state=pending>
parsl.dataflow.dflow DEBUG: Task 134 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 135 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: <AppFuture at 0x14aa199318d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 836c910399351fba796b5fa3b59e95ce
parsl.dataflow.memoization INFO: Task 135 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a416c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 46
parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/e03210b7-a72f-486f-9e3c-7c1a23142d69_isr_855910_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/e03210b7-a72f-486f-9e3c-7c1a23142d69_isr_855910_4.stderr
parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 136 submitted for App characterizeImage, waiting on task 135
parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: <AppFuture at 0x14aa19930850 state=pending>
parsl.dataflow.dflow DEBUG: Task 136 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 137 submitted for App calibrate, waiting on task 136
parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: <AppFuture at 0x14aa19932890 state=pending>
parsl.dataflow.dflow DEBUG: Task 137 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 138 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: <AppFuture at 0x14aa19931590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 5156c9c563c478edee5410b508e50f7e
parsl.dataflow.memoization INFO: Task 138 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a419e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 47
parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/fd3492ab-a3a9-4a2d-95b5-7b2be7672773_isr_855820_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/fd3492ab-a3a9-4a2d-95b5-7b2be7672773_isr_855820_47.stderr
parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 139 submitted for App characterizeImage, waiting on task 138
parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: <AppFuture at 0x14aa19932490 state=pending>
parsl.dataflow.dflow DEBUG: Task 139 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 140 submitted for App calibrate, waiting on task 139
parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: <AppFuture at 0x14aa18fcd6d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 140 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 141 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: <AppFuture at 0x14aa18fceb90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 141 has memoization hash d7f44461bc47f85d3206462644403b09
parsl.dataflow.memoization INFO: Task 141 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a41d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 48
parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/832fd814-a0e1-4a16-b7cc-3e83e1086855_isr_855821_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/832fd814-a0e1-4a16-b7cc-3e83e1086855_isr_855821_30.stderr
parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 142 submitted for App characterizeImage, waiting on task 141
parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: <AppFuture at 0x14aa18fcd090 state=pending>
parsl.dataflow.dflow DEBUG: Task 142 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 143 submitted for App calibrate, waiting on task 142
parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: <AppFuture at 0x14aa18fcfed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 143 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 144 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: <AppFuture at 0x14aa185fe390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 144 has memoization hash bbd2cd797e781871890c20d5fa504752
parsl.dataflow.memoization INFO: Task 144 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a42020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 49
parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/7603e76f-23e0-4c0d-ad21-920bc8b270a3_isr_855910_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/7603e76f-23e0-4c0d-ad21-920bc8b270a3_isr_855910_34.stderr
parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 145 submitted for App characterizeImage, waiting on task 144
parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: <AppFuture at 0x14aa1744ae50 state=pending>
parsl.dataflow.dflow DEBUG: Task 145 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 146 submitted for App calibrate, waiting on task 145
parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: <AppFuture at 0x14aa18aaaa90 state=pending>
parsl.dataflow.dflow DEBUG: Task 146 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 147 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: <AppFuture at 0x14aa18aaa290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 147 has memoization hash f56780f70044465f6f76116c3da76edb
parsl.dataflow.memoization INFO: Task 147 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a42340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 50
parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/48ef93b4-3555-465c-84f3-9b57e3cc6e4b_isr_855822_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/48ef93b4-3555-465c-84f3-9b57e3cc6e4b_isr_855822_23.stderr
parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 148 submitted for App characterizeImage, waiting on task 147
parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: <AppFuture at 0x14aa18aa9c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 148 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 149 submitted for App calibrate, waiting on task 148
parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: <AppFuture at 0x14aa18aab3d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 149 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 150 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: <AppFuture at 0x14aa182ebd50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 3273fb93c26ce1c3d394e02db55464a6
parsl.dataflow.memoization INFO: Task 150 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a42660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 51
parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/749780eb-505f-4244-a361-18678705070f_isr_855911_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/749780eb-505f-4244-a361-18678705070f_isr_855911_41.stderr
parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 151 submitted for App characterizeImage, waiting on task 150
parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: <AppFuture at 0x14aa182e85d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 151 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 152 submitted for App calibrate, waiting on task 151
parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: <AppFuture at 0x14aa187e6f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 152 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 153 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: <AppFuture at 0x14aa187e4d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 153 has memoization hash ad8fb20d1fc4807d6f72c3286d3e043e
parsl.dataflow.memoization INFO: Task 153 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a42980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 52
parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a0121ba7-0d79-4c5f-be04-7cbdb69d8e3a_isr_855911_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a0121ba7-0d79-4c5f-be04-7cbdb69d8e3a_isr_855911_37.stderr
parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 154 submitted for App characterizeImage, waiting on task 153
parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: <AppFuture at 0x14aa187e43d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 154 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 155 submitted for App calibrate, waiting on task 154
parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: <AppFuture at 0x14aa187e7d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 155 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 156 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: <AppFuture at 0x14aa17c46190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 9cbd0a887d7aeed7356c073270a89ada
parsl.dataflow.memoization INFO: Task 156 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a42ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 53
parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/79182091-7034-4fb2-a6e1-df4aabd3abb2_isr_855909_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/79182091-7034-4fb2-a6e1-df4aabd3abb2_isr_855909_12.stderr
parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 157 submitted for App characterizeImage, waiting on task 156
parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: <AppFuture at 0x14aa19b08c10 state=pending>
parsl.dataflow.dflow DEBUG: Task 157 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 158 submitted for App calibrate, waiting on task 157
parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: <AppFuture at 0x14aa1862bb50 state=pending>
parsl.dataflow.dflow DEBUG: Task 158 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 159 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: <AppFuture at 0x14aa18629910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 35ed5ede2ed7c917dcd7668328325ffe
parsl.dataflow.memoization INFO: Task 159 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a42fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 54
parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/f52fd248-4d46-48ad-acca-12fbda1b83cb_isr_855820_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/f52fd248-4d46-48ad-acca-12fbda1b83cb_isr_855820_1.stderr
parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 160 submitted for App characterizeImage, waiting on task 159
parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: <AppFuture at 0x14aa18628910 state=pending>
parsl.dataflow.dflow DEBUG: Task 160 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 161 submitted for App calibrate, waiting on task 160
parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: <AppFuture at 0x14aa186230d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 161 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 162 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: <AppFuture at 0x14aa185ba510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 9b5d77f7318b1eecc51ac03bae050e7d
parsl.dataflow.memoization INFO: Task 162 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a432e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 55
parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/34f02749-9576-4581-9a5f-a0bd160b8d5c_isr_855820_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/34f02749-9576-4581-9a5f-a0bd160b8d5c_isr_855820_26.stderr
parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 163 submitted for App characterizeImage, waiting on task 162
parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: <AppFuture at 0x14aa186210d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 163 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 164 submitted for App calibrate, waiting on task 163
parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: <AppFuture at 0x14aa183f5410 state=pending>
parsl.dataflow.dflow DEBUG: Task 164 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 165 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: <AppFuture at 0x14aa183f6410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 004628d512f427f5c536f061e1283c9a
parsl.dataflow.memoization INFO: Task 165 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a43600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 56
parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/3f667c69-86fb-4dfb-a8b5-323d038fa19e_isr_855822_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/3f667c69-86fb-4dfb-a8b5-323d038fa19e_isr_855822_25.stderr
parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 166 submitted for App characterizeImage, waiting on task 165
parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: <AppFuture at 0x14aa184191d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 166 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 167 submitted for App calibrate, waiting on task 166
parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: <AppFuture at 0x14aa1841add0 state=pending>
parsl.dataflow.dflow DEBUG: Task 167 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 168 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: <AppFuture at 0x14aa19925890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 168 has memoization hash ec180dc5c35eadb124205678a7d28fa8
parsl.dataflow.memoization INFO: Task 168 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a43920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 57
parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/fff4024b-5906-4bb6-8081-44839be3539b_isr_855909_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/fff4024b-5906-4bb6-8081-44839be3539b_isr_855909_43.stderr
parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 169 submitted for App characterizeImage, waiting on task 168
parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: <AppFuture at 0x14aa18418f50 state=pending>
parsl.dataflow.dflow DEBUG: Task 169 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 170 submitted for App calibrate, waiting on task 169
parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: <AppFuture at 0x14aa19927610 state=pending>
parsl.dataflow.dflow DEBUG: Task 170 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 171 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: <AppFuture at 0x14aa19927e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 171 has memoization hash 8f3aed92e4af93734e6efbc7f8ddedbd
parsl.dataflow.memoization INFO: Task 171 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a43c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 58
parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/e554d0ba-a1e1-44e6-a4e1-193ec1cb3a23_isr_855820_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/e554d0ba-a1e1-44e6-a4e1-193ec1cb3a23_isr_855820_37.stderr
parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 172 submitted for App characterizeImage, waiting on task 171
parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: <AppFuture at 0x14aa19927a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 172 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 173 submitted for App calibrate, waiting on task 172
parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: <AppFuture at 0x14aa19b14d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 173 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 174 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: <AppFuture at 0x14aa17933ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 174 has memoization hash f83d3ee443c5fdd59a29293fb17c4b5b
parsl.dataflow.memoization INFO: Task 174 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a43f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 59
parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/3f5c447f-f558-4219-82ed-56cc3791e0bf_isr_855822_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/3f5c447f-f558-4219-82ed-56cc3791e0bf_isr_855822_57.stderr
parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 175 submitted for App characterizeImage, waiting on task 174
parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: <AppFuture at 0x14aa18232bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 175 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 176 submitted for App calibrate, waiting on task 175
parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: <AppFuture at 0x14aa19af0450 state=pending>
parsl.dataflow.dflow DEBUG: Task 176 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 177 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: <AppFuture at 0x14aa19af1950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 177 has memoization hash e9506cbe2f11da0c31611fe6bc22650a
parsl.dataflow.memoization INFO: Task 177 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a642c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 60
parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/1fe0b93f-672b-4c3a-9a99-5e446c9fefc3_isr_855821_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/1fe0b93f-672b-4c3a-9a99-5e446c9fefc3_isr_855821_56.stderr
parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 178 submitted for App characterizeImage, waiting on task 177
parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: <AppFuture at 0x14aa19af0310 state=pending>
parsl.dataflow.dflow DEBUG: Task 178 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 179 submitted for App calibrate, waiting on task 178
parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: <AppFuture at 0x14aa19af3250 state=pending>
parsl.dataflow.dflow DEBUG: Task 179 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 180 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: <AppFuture at 0x14aa19af3bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 180 has memoization hash c1b18c0b458354117fc9318a22053b6a
parsl.dataflow.memoization INFO: Task 180 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a645e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 61
parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/e227bbd7-306a-41ea-9a80-8ed432204214_isr_855821_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/e227bbd7-306a-41ea-9a80-8ed432204214_isr_855821_36.stderr
parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 181 submitted for App characterizeImage, waiting on task 180
parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: <AppFuture at 0x14aa19af1550 state=pending>
parsl.dataflow.dflow DEBUG: Task 181 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 182 submitted for App calibrate, waiting on task 181
parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: <AppFuture at 0x14aa19af2b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 182 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 183 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: <AppFuture at 0x14aa19af0d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 174e89e6d237eaecb54e42c2f9d1766e
parsl.dataflow.memoization INFO: Task 183 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a64900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 62
parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/c49b272e-0b97-499d-bd6a-99abd921e485_isr_855821_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/c49b272e-0b97-499d-bd6a-99abd921e485_isr_855821_58.stderr
parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 184 submitted for App characterizeImage, waiting on task 183
parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: <AppFuture at 0x14aa183fd310 state=pending>
parsl.dataflow.dflow DEBUG: Task 184 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 185 submitted for App calibrate, waiting on task 184
parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: <AppFuture at 0x14aa183fd1d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 185 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 186 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: <AppFuture at 0x14aa17fdc050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 72d8436b9bc9a2f2b9b5e15e419229af
parsl.dataflow.memoization INFO: Task 186 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a64c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 63
parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/d1d2a5d7-97d9-4409-9159-2470bb3f092d_isr_855911_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/d1d2a5d7-97d9-4409-9159-2470bb3f092d_isr_855911_29.stderr
parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 187 submitted for App characterizeImage, waiting on task 186
parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: <AppFuture at 0x14aa17fdfa50 state=pending>
parsl.dataflow.dflow DEBUG: Task 187 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 188 submitted for App calibrate, waiting on task 187
parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: <AppFuture at 0x14aa17fdf850 state=pending>
parsl.dataflow.dflow DEBUG: Task 188 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 189 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: <AppFuture at 0x14aa17f639d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 189 has memoization hash 6f42ea689036f94223bd9d7797de077e
parsl.dataflow.memoization INFO: Task 189 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a64f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 64
parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/900274fa-504c-45ea-b55a-3c751c69723a_isr_855820_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/900274fa-504c-45ea-b55a-3c751c69723a_isr_855820_29.stderr
parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 190 submitted for App characterizeImage, waiting on task 189
parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: <AppFuture at 0x14aa17f63c10 state=pending>
parsl.dataflow.dflow DEBUG: Task 190 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 191 submitted for App calibrate, waiting on task 190
parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: <AppFuture at 0x14aa17bb2090 state=pending>
parsl.dataflow.dflow DEBUG: Task 191 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 192 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: <AppFuture at 0x14aa17bb3a50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 192 has memoization hash 34ea81b45f19c96efcf2a45dadcbf72f
parsl.dataflow.memoization INFO: Task 192 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a65260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 65
parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/4d6a5155-dcff-4461-b6ff-044929cbae98_isr_855822_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/4d6a5155-dcff-4461-b6ff-044929cbae98_isr_855822_60.stderr
parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 193 submitted for App characterizeImage, waiting on task 192
parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: <AppFuture at 0x14aa17bb1ad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 193 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 194 submitted for App calibrate, waiting on task 193
parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: <AppFuture at 0x14aa1784b450 state=pending>
parsl.dataflow.dflow DEBUG: Task 194 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 195 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: <AppFuture at 0x14aa17d369d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 2f5251661c60eb6a330982c924b3d2ec
parsl.dataflow.memoization INFO: Task 195 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a65580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 66
parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/e45fa78f-748c-455a-b7fd-a46298f0f6ae_isr_855822_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/e45fa78f-748c-455a-b7fd-a46298f0f6ae_isr_855822_59.stderr
parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 196 submitted for App characterizeImage, waiting on task 195
parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: <AppFuture at 0x14aa17e4bed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 196 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 197 submitted for App calibrate, waiting on task 196
parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: <AppFuture at 0x14aa17d34410 state=pending>
parsl.dataflow.dflow DEBUG: Task 197 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 198 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: <AppFuture at 0x14aa17c2db90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 60b7d32becd9d99cd1fac6abc4e7ac0e
parsl.dataflow.memoization INFO: Task 198 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a658a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 67
parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/04fd684c-3ae9-4228-9cfd-4e9e7d720881_isr_855910_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/04fd684c-3ae9-4228-9cfd-4e9e7d720881_isr_855910_15.stderr
parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 199 submitted for App characterizeImage, waiting on task 198
parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: <AppFuture at 0x14aa17d37010 state=pending>
parsl.dataflow.dflow DEBUG: Task 199 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 200 submitted for App calibrate, waiting on task 199
parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: <AppFuture at 0x14aa17b29210 state=pending>
parsl.dataflow.dflow DEBUG: Task 200 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 201 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: <AppFuture at 0x14aa17b2a1d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 847bfbe2d00e2a2dfee010615fe069ff
parsl.dataflow.memoization INFO: Task 201 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a65bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 68
parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/b3f9a143-6f60-471b-a52f-81284361f8c2_isr_855911_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/b3f9a143-6f60-471b-a52f-81284361f8c2_isr_855911_2.stderr
parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 202 submitted for App characterizeImage, waiting on task 201
parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: <AppFuture at 0x14aa17b59f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 202 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 203 submitted for App calibrate, waiting on task 202
parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: <AppFuture at 0x14aa17a8eb90 state=pending>
parsl.dataflow.dflow DEBUG: Task 203 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 204 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: <AppFuture at 0x14aa17a8cbd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 788b9532b301cfe477c203eebe8b951c
parsl.dataflow.memoization INFO: Task 204 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a65ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 69
parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/b83c54be-4ac7-4baa-b4b8-39fcb809ac4a_isr_855910_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/b83c54be-4ac7-4baa-b4b8-39fcb809ac4a_isr_855910_49.stderr
parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 205 submitted for App characterizeImage, waiting on task 204
parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: <AppFuture at 0x14aa17a8ca50 state=pending>
parsl.dataflow.dflow DEBUG: Task 205 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 206 submitted for App calibrate, waiting on task 205
parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: <AppFuture at 0x14aa1795d550 state=pending>
parsl.dataflow.dflow DEBUG: Task 206 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 207 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: <AppFuture at 0x14aa179f4190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 207 has memoization hash f117d25131d3e1aed3b97b09d6946f2d
parsl.dataflow.memoization INFO: Task 207 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a66200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 70
parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/c978c519-2d2d-4b1e-8d44-1cb35ebfd8a9_isr_855911_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/c978c519-2d2d-4b1e-8d44-1cb35ebfd8a9_isr_855911_4.stderr
parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 208 submitted for App characterizeImage, waiting on task 207
parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: <AppFuture at 0x14aa18fcdfd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 208 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 209 submitted for App calibrate, waiting on task 208
parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: <AppFuture at 0x14aa178ff210 state=pending>
parsl.dataflow.dflow DEBUG: Task 209 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 210 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: <AppFuture at 0x14aa19ae0190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 65b8a0378a15bcbf4aa56cc0447a67fd
parsl.dataflow.memoization INFO: Task 210 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a66520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 71
parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/298f3187-2d40-4eef-8c3f-5510bfe7624b_isr_855909_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/298f3187-2d40-4eef-8c3f-5510bfe7624b_isr_855909_9.stderr
parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 211 submitted for App characterizeImage, waiting on task 210
parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: <AppFuture at 0x14aa178fce90 state=pending>
parsl.dataflow.dflow DEBUG: Task 211 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 212 submitted for App calibrate, waiting on task 211
parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: <AppFuture at 0x14aa19ae1cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 212 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 213 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: <AppFuture at 0x14aa19ae2610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 6def333686093877731d4fd9c53eb41c
parsl.dataflow.memoization INFO: Task 213 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a66840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 72
parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/08242780-7300-46c2-b518-ea60416ae3ba_isr_855820_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/08242780-7300-46c2-b518-ea60416ae3ba_isr_855820_18.stderr
parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 214 submitted for App characterizeImage, waiting on task 213
parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: <AppFuture at 0x14aa19ae2b90 state=pending>
parsl.dataflow.dflow DEBUG: Task 214 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 215 submitted for App calibrate, waiting on task 214
parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: <AppFuture at 0x14aa19ae0e50 state=pending>
parsl.dataflow.dflow DEBUG: Task 215 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 216 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: <AppFuture at 0x14aa19ae3bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 216 has memoization hash 849d08d123ddb31f04135ed75483fd43
parsl.dataflow.memoization INFO: Task 216 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a66b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 73
parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/b6615050-2244-40e6-b26c-d50b9d9df275_isr_855820_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/b6615050-2244-40e6-b26c-d50b9d9df275_isr_855820_13.stderr
parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 217 submitted for App characterizeImage, waiting on task 216
parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: <AppFuture at 0x14aa19ae2d50 state=pending>
parsl.dataflow.dflow DEBUG: Task 217 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 218 submitted for App calibrate, waiting on task 217
parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: <AppFuture at 0x14aa174666d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 218 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 219 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: <AppFuture at 0x14aa172c0a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 1e4030f2e8dd9cebb678f2f2a30d3a3a
parsl.dataflow.memoization INFO: Task 219 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a66e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 74
parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/70205c51-8fbb-4b29-a7b6-c7d07e53c86f_isr_855822_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/70205c51-8fbb-4b29-a7b6-c7d07e53c86f_isr_855822_6.stderr
parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 220 submitted for App characterizeImage, waiting on task 219
parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: <AppFuture at 0x14aa172c2110 state=pending>
parsl.dataflow.dflow DEBUG: Task 220 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 221 submitted for App calibrate, waiting on task 220
parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: <AppFuture at 0x14aa172c37d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 221 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 222 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: <AppFuture at 0x14aa17367b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 222 has memoization hash 37acff8635e4a7bed0a76de2cefbdc68
parsl.dataflow.memoization INFO: Task 222 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a671a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 75
parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/7d67734a-124e-44eb-bb94-dee89d6d1a5d_isr_855821_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/7d67734a-124e-44eb-bb94-dee89d6d1a5d_isr_855821_37.stderr
parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 223 submitted for App characterizeImage, waiting on task 222
parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: <AppFuture at 0x14aa17367690 state=pending>
parsl.dataflow.dflow DEBUG: Task 223 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 224 submitted for App calibrate, waiting on task 223
parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: <AppFuture at 0x14aa173fcc10 state=pending>
parsl.dataflow.dflow DEBUG: Task 224 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 225 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: <AppFuture at 0x14aa173d6190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 225 has memoization hash e9f486b730958364fc00a00a8a99d050
parsl.dataflow.memoization INFO: Task 225 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a674c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 76
parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/6fc76c45-64ad-41c3-8f11-f5958d29a3a1_isr_855821_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/6fc76c45-64ad-41c3-8f11-f5958d29a3a1_isr_855821_33.stderr
parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 226 submitted for App characterizeImage, waiting on task 225
parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: <AppFuture at 0x14aa17164a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 226 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 227 submitted for App calibrate, waiting on task 226
parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: <AppFuture at 0x14aa19adad10 state=pending>
parsl.dataflow.dflow DEBUG: Task 227 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 228 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: <AppFuture at 0x14aa19adb550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 228 has memoization hash 6ed99c68f68deb045be31f4362b180d9
parsl.dataflow.memoization INFO: Task 228 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a677e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 77
parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/a7d09716-7d80-47c1-b7ae-95a56c7a83a7_isr_855822_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/a7d09716-7d80-47c1-b7ae-95a56c7a83a7_isr_855822_39.stderr
parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 229 submitted for App characterizeImage, waiting on task 228
parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: <AppFuture at 0x14aa19ada7d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 229 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 230 submitted for App calibrate, waiting on task 229
parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: <AppFuture at 0x14aa19ad9210 state=pending>
parsl.dataflow.dflow DEBUG: Task 230 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 231 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: <AppFuture at 0x14aa19ad8710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 231 has memoization hash 77da93fb77cdb75ce3ab8909df0a3e3f
parsl.dataflow.memoization INFO: Task 231 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a67b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 78
parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/16b58f6c-d4fd-429e-a436-887720a104e7_isr_855909_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/16b58f6c-d4fd-429e-a436-887720a104e7_isr_855909_26.stderr
parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 232 submitted for App characterizeImage, waiting on task 231
parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: <AppFuture at 0x14aa19ada110 state=pending>
parsl.dataflow.dflow DEBUG: Task 232 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 233 submitted for App calibrate, waiting on task 232
parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: <AppFuture at 0x14aa19ada810 state=pending>
parsl.dataflow.dflow DEBUG: Task 233 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 234 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: <AppFuture at 0x14aa18b990d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 234 has memoization hash d155b368ee4a4dff223a963d8360991b
parsl.dataflow.memoization INFO: Task 234 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a67e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 79
parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/3f88f97f-346f-4413-923f-af5780e90136_isr_855821_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/3f88f97f-346f-4413-923f-af5780e90136_isr_855821_31.stderr
parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 235 submitted for App characterizeImage, waiting on task 234
parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: <AppFuture at 0x14aa18b9b490 state=pending>
parsl.dataflow.dflow DEBUG: Task 235 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 236 submitted for App calibrate, waiting on task 235
parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: <AppFuture at 0x14aa18b9bed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 236 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 237 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: <AppFuture at 0x14aa18b9a7d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 237 has memoization hash d8ac6bd5c8e0d3b22ca60a84e4945914
parsl.dataflow.memoization INFO: Task 237 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 80
parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/ac1ee20a-9d1c-4e75-bfd8-853a271fbe0d_isr_855820_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/ac1ee20a-9d1c-4e75-bfd8-853a271fbe0d_isr_855820_2.stderr
parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 238 submitted for App characterizeImage, waiting on task 237
parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: <AppFuture at 0x14aa18b98cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 238 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 239 submitted for App calibrate, waiting on task 238
parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: <AppFuture at 0x14aa173f1750 state=pending>
parsl.dataflow.dflow DEBUG: Task 239 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 240 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: <AppFuture at 0x14aa1991fd90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 9e6131faa9da0dc00b068350bfbcd4c7
parsl.dataflow.memoization INFO: Task 240 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 81
parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/8414eacf-78a0-4725-8534-e2d8ff9d78a1_isr_855822_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/8414eacf-78a0-4725-8534-e2d8ff9d78a1_isr_855822_20.stderr
parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 241 submitted for App characterizeImage, waiting on task 240
parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: <AppFuture at 0x14aa1991c190 state=pending>
parsl.dataflow.dflow DEBUG: Task 241 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 242 submitted for App calibrate, waiting on task 241
parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: <AppFuture at 0x14aa1991cfd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 242 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 243 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: <AppFuture at 0x14aa1991c150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 243 has memoization hash 87a74bcf96efbff4c5aef5d4099f71b9
parsl.dataflow.memoization INFO: Task 243 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 82
parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/4fdfc300-9543-4666-9c9d-75ee1ed00eaa_isr_855820_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/4fdfc300-9543-4666-9c9d-75ee1ed00eaa_isr_855820_59.stderr
parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 244 submitted for App characterizeImage, waiting on task 243
parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: <AppFuture at 0x14aa1991c810 state=pending>
parsl.dataflow.dflow DEBUG: Task 244 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 245 submitted for App calibrate, waiting on task 244
parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: <AppFuture at 0x14aa18f45990 state=pending>
parsl.dataflow.dflow DEBUG: Task 245 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 246 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: <AppFuture at 0x14aa18f46b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 246 has memoization hash e4cd8e62774ad5c12e9005926c3b7869
parsl.dataflow.memoization INFO: Task 246 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 83
parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/40e896a3-b962-4cef-b188-e209f28ee26b_isr_855909_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/40e896a3-b962-4cef-b188-e209f28ee26b_isr_855909_29.stderr
parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 247 submitted for App characterizeImage, waiting on task 246
parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: <AppFuture at 0x14aa18f45d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 247 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 248 submitted for App calibrate, waiting on task 247
parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: <AppFuture at 0x14aa18f46110 state=pending>
parsl.dataflow.dflow DEBUG: Task 248 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 249 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: <AppFuture at 0x14aa17115b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 249 has memoization hash 438f62b25463505aeaefd6df4561fa0d
parsl.dataflow.memoization INFO: Task 249 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 84
parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/f5601c98-96f5-4974-9611-d091f863f617_isr_855822_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/f5601c98-96f5-4974-9611-d091f863f617_isr_855822_15.stderr
parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 250 submitted for App characterizeImage, waiting on task 249
parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: <AppFuture at 0x14aa171148d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 250 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 251 submitted for App calibrate, waiting on task 250
parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: <AppFuture at 0x14aa1993c3d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 251 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 252 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: <AppFuture at 0x14aa1993d5d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 252 has memoization hash 8a95dba507e9084065923a36eb6f040b
parsl.dataflow.memoization INFO: Task 252 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 85
parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/c8aacc6c-64c7-4f89-a359-1df47c18d57d_isr_855910_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/c8aacc6c-64c7-4f89-a359-1df47c18d57d_isr_855910_7.stderr
parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 253 submitted for App characterizeImage, waiting on task 252
parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: <AppFuture at 0x14aa1993e150 state=pending>
parsl.dataflow.dflow DEBUG: Task 253 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 254 submitted for App calibrate, waiting on task 253
parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: <AppFuture at 0x14aa1993e090 state=pending>
parsl.dataflow.dflow DEBUG: Task 254 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 255 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: <AppFuture at 0x14aa18b79310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 255 has memoization hash 2d17026c4e91a60069ddf3d31192869d
parsl.dataflow.memoization INFO: Task 255 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 86
parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/afe62168-58bf-461a-a93a-910a2b626907_isr_855820_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/afe62168-58bf-461a-a93a-910a2b626907_isr_855820_33.stderr
parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 256 submitted for App characterizeImage, waiting on task 255
parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: <AppFuture at 0x14aa1993c850 state=pending>
parsl.dataflow.dflow DEBUG: Task 256 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 257 submitted for App calibrate, waiting on task 256
parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: <AppFuture at 0x14aa18b79ed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 257 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 258 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: <AppFuture at 0x14aa17132590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 258 has memoization hash 761d7c527345e72464d2a030660537ac
parsl.dataflow.memoization INFO: Task 258 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 87
parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/9f572600-2953-4721-9d09-60c594a9f0d5_isr_855821_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/9f572600-2953-4721-9d09-60c594a9f0d5_isr_855821_38.stderr
parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 259 submitted for App characterizeImage, waiting on task 258
parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: <AppFuture at 0x14aa18b78d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 259 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 260 submitted for App calibrate, waiting on task 259
parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: <AppFuture at 0x14aa18df0850 state=pending>
parsl.dataflow.dflow DEBUG: Task 260 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 261 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: <AppFuture at 0x14aa19d574d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 261 has memoization hash 065be44d650c33b2de381b55514efe30
parsl.dataflow.memoization INFO: Task 261 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 88
parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0686c199-db34-403f-aec5-3d7290adf814_isr_855820_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0686c199-db34-403f-aec5-3d7290adf814_isr_855820_25.stderr
parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 262 submitted for App characterizeImage, waiting on task 261
parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: <AppFuture at 0x14aa19d55d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 262 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 263 submitted for App calibrate, waiting on task 262
parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: <AppFuture at 0x14aa19d56050 state=pending>
parsl.dataflow.dflow DEBUG: Task 263 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 264 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: <AppFuture at 0x14aa19d57ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 264 has memoization hash 307b9bb7d532732c39cf51bf05795553
parsl.dataflow.memoization INFO: Task 264 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 89
parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/a38e9d92-e7a9-4662-94c0-502d800146d9_isr_855909_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/a38e9d92-e7a9-4662-94c0-502d800146d9_isr_855909_22.stderr
parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 265 submitted for App characterizeImage, waiting on task 264
parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: <AppFuture at 0x14aa19d56bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 265 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 266 submitted for App calibrate, waiting on task 265
parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: <AppFuture at 0x14aa19d569d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 266 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 267 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: <AppFuture at 0x14aa19d57090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 267 has memoization hash c8ee658467af959038342d0c5ca30d3f
parsl.dataflow.memoization INFO: Task 267 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 90
parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/f9edba97-d5fe-41d9-a1ae-d0e5dfc4b312_isr_855820_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/f9edba97-d5fe-41d9-a1ae-d0e5dfc4b312_isr_855820_55.stderr
parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 268 submitted for App characterizeImage, waiting on task 267
parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: <AppFuture at 0x14aa19ebae50 state=pending>
parsl.dataflow.dflow DEBUG: Task 268 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 269 submitted for App calibrate, waiting on task 268
parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: <AppFuture at 0x14aa19ebb910 state=pending>
parsl.dataflow.dflow DEBUG: Task 269 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 270 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: <AppFuture at 0x14aa19d4d010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 270 has memoization hash 46d4152b1850d1f77e7fce04efd72f93
parsl.dataflow.memoization INFO: Task 270 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 91
parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/62f46889-646e-4827-9fff-99e67ee588cd_isr_855909_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/62f46889-646e-4827-9fff-99e67ee588cd_isr_855909_7.stderr
parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 271 submitted for App characterizeImage, waiting on task 270
parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: <AppFuture at 0x14aa19d4d350 state=pending>
parsl.dataflow.dflow DEBUG: Task 271 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 272 submitted for App calibrate, waiting on task 271
parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: <AppFuture at 0x14aa19d4e550 state=pending>
parsl.dataflow.dflow DEBUG: Task 272 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 273 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: <AppFuture at 0x14aa176713d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 273 has memoization hash 95d718212725e58b55316f495456c8ba
parsl.dataflow.memoization INFO: Task 273 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 92
parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/8b0c070e-1b36-46b5-9c12-2434db50fc58_isr_855910_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/8b0c070e-1b36-46b5-9c12-2434db50fc58_isr_855910_52.stderr
parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 274 submitted for App characterizeImage, waiting on task 273
parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: <AppFuture at 0x14aa176723d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 274 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 275 submitted for App calibrate, waiting on task 274
parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: <AppFuture at 0x14aa19d43510 state=pending>
parsl.dataflow.dflow DEBUG: Task 275 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 276 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: <AppFuture at 0x14aa19d40e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 276 has memoization hash bc03486a6d82701207ed5d69cd79a2b9
parsl.dataflow.memoization INFO: Task 276 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 93
parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a924cadd-4789-48e5-bf78-7a16efcc32d5_isr_855911_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a924cadd-4789-48e5-bf78-7a16efcc32d5_isr_855911_34.stderr
parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 277 submitted for App characterizeImage, waiting on task 276
parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: <AppFuture at 0x14aa19d41110 state=pending>
parsl.dataflow.dflow DEBUG: Task 277 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 278 submitted for App calibrate, waiting on task 277
parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: <AppFuture at 0x14aa19d41c10 state=pending>
parsl.dataflow.dflow DEBUG: Task 278 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 279 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: <AppFuture at 0x14aa19d43b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 279 has memoization hash 143790abd175443e706e0c02e130e6f6
parsl.dataflow.memoization INFO: Task 279 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 94
parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/35ba0be6-1152-488c-93bf-a42ee0243dbe_isr_855821_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/35ba0be6-1152-488c-93bf-a42ee0243dbe_isr_855821_46.stderr
parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 280 submitted for App characterizeImage, waiting on task 279
parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: <AppFuture at 0x14aa19d43850 state=pending>
parsl.dataflow.dflow DEBUG: Task 280 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 281 submitted for App calibrate, waiting on task 280
parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: <AppFuture at 0x14aa19edf250 state=pending>
parsl.dataflow.dflow DEBUG: Task 281 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 282 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: <AppFuture at 0x14aa19edee10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 282 has memoization hash 587d74c0296596eb17e4530d10b1f74d
parsl.dataflow.memoization INFO: Task 282 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 95
parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/9491ef52-c0bb-4376-a38f-825a617ac659_isr_855910_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/9491ef52-c0bb-4376-a38f-825a617ac659_isr_855910_41.stderr
parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 283 submitted for App characterizeImage, waiting on task 282
parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: <AppFuture at 0x14aa19ede390 state=pending>
parsl.dataflow.dflow DEBUG: Task 283 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 284 submitted for App calibrate, waiting on task 283
parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: <AppFuture at 0x14aa19edea50 state=pending>
parsl.dataflow.dflow DEBUG: Task 284 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 285 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: <AppFuture at 0x14aa176f2a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 285 has memoization hash bb13296f9ac2ab57568f0cd0883a9e94
parsl.dataflow.memoization INFO: Task 285 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 96
parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/a80963c7-fa46-4ca0-b8f5-055c94f4bd09_isr_855822_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/a80963c7-fa46-4ca0-b8f5-055c94f4bd09_isr_855822_46.stderr
parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 286 submitted for App characterizeImage, waiting on task 285
parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: <AppFuture at 0x14aa19edefd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 286 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 287 submitted for App calibrate, waiting on task 286
parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: <AppFuture at 0x14aa176f0d50 state=pending>
parsl.dataflow.dflow DEBUG: Task 287 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 288 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: <AppFuture at 0x14aa17a3e210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 288 has memoization hash 5f0a512231637ba25b0be0b385ba80cc
parsl.dataflow.memoization INFO: Task 288 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 97
parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/245ac0e3-5bbb-465c-ac89-8875f42abf4c_isr_855909_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/245ac0e3-5bbb-465c-ac89-8875f42abf4c_isr_855909_11.stderr
parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 289 submitted for App characterizeImage, waiting on task 288
parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: <AppFuture at 0x14aa19b17e50 state=pending>
parsl.dataflow.dflow DEBUG: Task 289 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 290 submitted for App calibrate, waiting on task 289
parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: <AppFuture at 0x14aa1702c290 state=pending>
parsl.dataflow.dflow DEBUG: Task 290 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 291 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: <AppFuture at 0x14aa19cdf8d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 291 has memoization hash 2c3ee8ace56d6227c11044322f428631
parsl.dataflow.memoization INFO: Task 291 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 98
parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/31f20752-eeeb-4be3-8ee0-b2ccd9911b1d_isr_855911_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/31f20752-eeeb-4be3-8ee0-b2ccd9911b1d_isr_855911_14.stderr
parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 292 submitted for App characterizeImage, waiting on task 291
parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: <AppFuture at 0x14aa19cddf90 state=pending>
parsl.dataflow.dflow DEBUG: Task 292 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 293 submitted for App calibrate, waiting on task 292
parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: <AppFuture at 0x14aa19cde210 state=pending>
parsl.dataflow.dflow DEBUG: Task 293 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 294 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: <AppFuture at 0x14aa19cdc110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 294 has memoization hash 18adfbe3bcc40402b50b1c34864bb9e0
parsl.dataflow.memoization INFO: Task 294 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 99
parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/ddc9536e-a6c5-4f78-8f7c-4e28a9238e52_isr_855910_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/ddc9536e-a6c5-4f78-8f7c-4e28a9238e52_isr_855910_17.stderr
parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 295 submitted for App characterizeImage, waiting on task 294
parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: <AppFuture at 0x14aa17aeb450 state=pending>
parsl.dataflow.dflow DEBUG: Task 295 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 296 submitted for App calibrate, waiting on task 295
parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: <AppFuture at 0x14aa17ae8910 state=pending>
parsl.dataflow.dflow DEBUG: Task 296 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 297 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: <AppFuture at 0x14aa18e21b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 867b29ec6a5200ecd8a249478f938a26
parsl.dataflow.memoization INFO: Task 297 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 100
parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/4ef3fe48-deed-4335-99ea-11054d9f188d_isr_855821_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/4ef3fe48-deed-4335-99ea-11054d9f188d_isr_855821_18.stderr
parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 298 submitted for App characterizeImage, waiting on task 297
parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: <AppFuture at 0x14aa18e21950 state=pending>
parsl.dataflow.dflow DEBUG: Task 298 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 299 submitted for App calibrate, waiting on task 298
parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: <AppFuture at 0x14aa18e21590 state=pending>
parsl.dataflow.dflow DEBUG: Task 299 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 300 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: <AppFuture at 0x14aa17d56b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 300 has memoization hash 79ecf59551fb9d9065a5bbca7377e850
parsl.dataflow.memoization INFO: Task 300 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 101
parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/fcd2c3f3-7e80-45c5-865f-a05ecf7173e3_isr_855821_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/fcd2c3f3-7e80-45c5-865f-a05ecf7173e3_isr_855821_15.stderr
parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 301 submitted for App characterizeImage, waiting on task 300
parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: <AppFuture at 0x14aa17d57050 state=pending>
parsl.dataflow.dflow DEBUG: Task 301 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 302 submitted for App calibrate, waiting on task 301
parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: <AppFuture at 0x14aa17b46d50 state=pending>
parsl.dataflow.dflow DEBUG: Task 302 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 303 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: <AppFuture at 0x14aa18179c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 303 has memoization hash 97dea8dd8307977ea171a6f5c2c5f06b
parsl.dataflow.memoization INFO: Task 303 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 102
parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/095f7b42-f9c2-4c21-9eef-2c4822825944_isr_855909_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/095f7b42-f9c2-4c21-9eef-2c4822825944_isr_855909_5.stderr
parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 304 submitted for App characterizeImage, waiting on task 303
parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: <AppFuture at 0x14aa18179b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 304 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 305 submitted for App calibrate, waiting on task 304
parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: <AppFuture at 0x14aa1817bc10 state=pending>
parsl.dataflow.dflow DEBUG: Task 305 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 306 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: <AppFuture at 0x14aa1817b3d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 306 has memoization hash 6981e93ead115aa73a53749d111525ef
parsl.dataflow.memoization INFO: Task 306 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 103
parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/4e613f7e-ddcc-4fc4-8ce9-8a7ab65b5c1e_isr_855910_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/4e613f7e-ddcc-4fc4-8ce9-8a7ab65b5c1e_isr_855910_47.stderr
parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 307 submitted for App characterizeImage, waiting on task 306
parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: <AppFuture at 0x14aa1817a890 state=pending>
parsl.dataflow.dflow DEBUG: Task 307 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 308 submitted for App calibrate, waiting on task 307
parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: <AppFuture at 0x14aa18e8a450 state=pending>
parsl.dataflow.dflow DEBUG: Task 308 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 309 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: <AppFuture at 0x14aa17cc7d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 23ef97170ab3d05d87d8f717047f0955
parsl.dataflow.memoization INFO: Task 309 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 104
parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/1e9c58f7-eb17-4d1e-9098-e2e8a3a5015d_isr_855911_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/1e9c58f7-eb17-4d1e-9098-e2e8a3a5015d_isr_855911_21.stderr
parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 310 submitted for App characterizeImage, waiting on task 309
parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: <AppFuture at 0x14aa17cc6690 state=pending>
parsl.dataflow.dflow DEBUG: Task 310 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 311 submitted for App calibrate, waiting on task 310
parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: <AppFuture at 0x14aa178fe150 state=pending>
parsl.dataflow.dflow DEBUG: Task 311 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 312 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: <AppFuture at 0x14aa18e00790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 395f7cc10aba936c04a0f398542ef1c3
parsl.dataflow.memoization INFO: Task 312 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 105
parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/f34a1d8c-66cd-43ed-a810-1dc7d5ac4ab1_isr_855911_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/f34a1d8c-66cd-43ed-a810-1dc7d5ac4ab1_isr_855911_15.stderr
parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 313 submitted for App characterizeImage, waiting on task 312
parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: <AppFuture at 0x14aa18e03f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 313 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 314 submitted for App calibrate, waiting on task 313
parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: <AppFuture at 0x14aa18e02b10 state=pending>
parsl.dataflow.dflow DEBUG: Task 314 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 315 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: <AppFuture at 0x14aa18e02510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 315 has memoization hash cce3fda8964747abc04d0f32ebcca3e5
parsl.dataflow.memoization INFO: Task 315 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 106
parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/138afd5f-3c6e-482e-955b-30a436ede0e9_isr_855822_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/138afd5f-3c6e-482e-955b-30a436ede0e9_isr_855822_1.stderr
parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 316 submitted for App characterizeImage, waiting on task 315
parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: <AppFuture at 0x14aa18e01ed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 316 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 317 submitted for App calibrate, waiting on task 316
parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: <AppFuture at 0x14aa18e00b10 state=pending>
parsl.dataflow.dflow DEBUG: Task 317 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 318 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: <AppFuture at 0x14aa18e00590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 318 has memoization hash cbf94cd25cdacd1d7b68f370a2f1119c
parsl.dataflow.memoization INFO: Task 318 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 107
parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/c4ff138e-90b3-4230-b9a0-7ce78fcc90b2_isr_855909_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/c4ff138e-90b3-4230-b9a0-7ce78fcc90b2_isr_855909_36.stderr
parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 319 submitted for App characterizeImage, waiting on task 318
parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: <AppFuture at 0x14aa17ce52d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 319 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 320 submitted for App calibrate, waiting on task 319
parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: <AppFuture at 0x14aa18e00a90 state=pending>
parsl.dataflow.dflow DEBUG: Task 320 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 321 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: <AppFuture at 0x14aa17f11290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 321 has memoization hash fde7126bf517bad362e6ffaf50eb5830
parsl.dataflow.memoization INFO: Task 321 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 108
parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/7e5a8b03-65e3-4c90-9e1e-2c59fcd23dc5_isr_855909_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/7e5a8b03-65e3-4c90-9e1e-2c59fcd23dc5_isr_855909_17.stderr
parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 322 submitted for App characterizeImage, waiting on task 321
parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: <AppFuture at 0x14aa17ce7e10 state=pending>
parsl.dataflow.dflow DEBUG: Task 322 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 323 submitted for App calibrate, waiting on task 322
parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: <AppFuture at 0x14aa18bcf5d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 323 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 324 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: <AppFuture at 0x14aa18077fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 324 has memoization hash 2bc071bf1a7237b43b84d13c93b5e0a1
parsl.dataflow.memoization INFO: Task 324 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 109
parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a1f6eb89-6b84-4d84-b187-895c699c561f_isr_855911_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a1f6eb89-6b84-4d84-b187-895c699c561f_isr_855911_61.stderr
parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 325 submitted for App characterizeImage, waiting on task 324
parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: <AppFuture at 0x14aa18ac0d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 325 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 326 submitted for App calibrate, waiting on task 325
parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: <AppFuture at 0x14aa1734c3d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 326 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 327 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: <AppFuture at 0x14aa1734cf10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 327 has memoization hash 1f6b448f174fe448f900696e2678ad19
parsl.dataflow.memoization INFO: Task 327 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 110
parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/de7d2dba-dced-4027-81bf-f24e47f5c104_isr_855911_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/de7d2dba-dced-4027-81bf-f24e47f5c104_isr_855911_31.stderr
parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 328 submitted for App characterizeImage, waiting on task 327
parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: <AppFuture at 0x14aa1734d3d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 328 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 329 submitted for App calibrate, waiting on task 328
parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: <AppFuture at 0x14aa17bcc710 state=pending>
parsl.dataflow.dflow DEBUG: Task 329 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 330 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: <AppFuture at 0x14aa17bcc090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 5b3b0429f94a1fdbdc7fe7bd35c8be77
parsl.dataflow.memoization INFO: Task 330 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16aba2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 111
parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/b5b71f3b-85ef-4711-bc95-53a25f91d9cc_isr_855911_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/b5b71f3b-85ef-4711-bc95-53a25f91d9cc_isr_855911_23.stderr
parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 331 submitted for App characterizeImage, waiting on task 330
parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: <AppFuture at 0x14aa18bcd090 state=pending>
parsl.dataflow.dflow DEBUG: Task 331 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 332 submitted for App calibrate, waiting on task 331
parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: <AppFuture at 0x14aa17d27d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 332 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 333 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: <AppFuture at 0x14aa188c18d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 333 has memoization hash 37fea1429df7cbc9028861091e68b01f
parsl.dataflow.memoization INFO: Task 333 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16aba5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 112
parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/0193b4c6-7d2f-4e46-bf89-c0ef4beaf065_isr_855822_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/0193b4c6-7d2f-4e46-bf89-c0ef4beaf065_isr_855822_62.stderr
parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 334 submitted for App characterizeImage, waiting on task 333
parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: <AppFuture at 0x14aa18f45f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 334 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 335 submitted for App calibrate, waiting on task 334
parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: <AppFuture at 0x14aa17525450 state=pending>
parsl.dataflow.dflow DEBUG: Task 335 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 336 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: <AppFuture at 0x14aa186fe410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 336 has memoization hash 55f8a2ef4ad651507a0b8d67d2202c8b
parsl.dataflow.memoization INFO: Task 336 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16aba8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 113
parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/415346ad-35b0-43cb-ac63-e1058f7ecc2f_isr_855820_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/415346ad-35b0-43cb-ac63-e1058f7ecc2f_isr_855820_10.stderr
parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 337 submitted for App characterizeImage, waiting on task 336
parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: <AppFuture at 0x14aa186fce90 state=pending>
parsl.dataflow.dflow DEBUG: Task 337 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 338 submitted for App calibrate, waiting on task 337
parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: <AppFuture at 0x14aa186fe3d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 338 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 339 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: <AppFuture at 0x14aa1a555dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 339 has memoization hash d1318cf2f6aa6f6e5e9c72573b1a648f
parsl.dataflow.memoization INFO: Task 339 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16abac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 114
parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/ec877263-403d-4fd7-bfad-453559fc1baf_isr_855821_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/ec877263-403d-4fd7-bfad-453559fc1baf_isr_855821_13.stderr
parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 340 submitted for App characterizeImage, waiting on task 339
parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: <AppFuture at 0x14aa18e03950 state=pending>
parsl.dataflow.dflow DEBUG: Task 340 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 341 submitted for App calibrate, waiting on task 340
parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: <AppFuture at 0x14aa17af9d50 state=pending>
parsl.dataflow.dflow DEBUG: Task 341 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 342 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: <AppFuture at 0x14aa18376690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 342 has memoization hash ae30bc58307758fc2a125122643d20ea
parsl.dataflow.memoization INFO: Task 342 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16abaf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 115
parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/5c803b1f-c8d7-4978-97e4-104d602c4ed1_isr_855910_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/5c803b1f-c8d7-4978-97e4-104d602c4ed1_isr_855910_57.stderr
parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 343 submitted for App characterizeImage, waiting on task 342
parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: <AppFuture at 0x14aa18376c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 343 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 344 submitted for App calibrate, waiting on task 343
parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: <AppFuture at 0x14aa18377910 state=pending>
parsl.dataflow.dflow DEBUG: Task 344 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 345 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: <AppFuture at 0x14aa17a75fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 345 has memoization hash 400244d7102faf5c01bda8895c361a18
parsl.dataflow.memoization INFO: Task 345 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16abb240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 116
parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/7e9df86d-965c-4472-87bb-5b8650f28f02_isr_855910_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/7e9df86d-965c-4472-87bb-5b8650f28f02_isr_855910_27.stderr
parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 346 submitted for App characterizeImage, waiting on task 345
parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: <AppFuture at 0x14aa17a750d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 346 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 347 submitted for App calibrate, waiting on task 346
parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: <AppFuture at 0x14aa19cdd610 state=pending>
parsl.dataflow.dflow DEBUG: Task 347 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 348 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: <AppFuture at 0x14aa184f5f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 348 has memoization hash 9ec442ed9293ad106e7c054966754659
parsl.dataflow.memoization INFO: Task 348 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16abb560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 117
parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/8a7ab59f-dcea-408e-8797-a9e5a0053e21_isr_855909_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/8a7ab59f-dcea-408e-8797-a9e5a0053e21_isr_855909_25.stderr
parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 349 submitted for App characterizeImage, waiting on task 348
parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: <AppFuture at 0x14aa184f7390 state=pending>
parsl.dataflow.dflow DEBUG: Task 349 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 350 submitted for App calibrate, waiting on task 349
parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: <AppFuture at 0x14aa1b5ecf10 state=pending>
parsl.dataflow.dflow DEBUG: Task 350 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 351 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: <AppFuture at 0x14aa1b5ed950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 351 has memoization hash 22f682f6ab6ce5b820b46effcd8e74db
parsl.dataflow.memoization INFO: Task 351 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16abb880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 118
parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/a0fad8ea-dd3d-4057-a423-fb2f072527d3_isr_855821_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/a0fad8ea-dd3d-4057-a423-fb2f072527d3_isr_855821_21.stderr
parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 352 submitted for App characterizeImage, waiting on task 351
parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: <AppFuture at 0x14aa1b5ee090 state=pending>
parsl.dataflow.dflow DEBUG: Task 352 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 353 submitted for App calibrate, waiting on task 352
parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: <AppFuture at 0x14aa1b5ef450 state=pending>
parsl.dataflow.dflow DEBUG: Task 353 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 354 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: <AppFuture at 0x14aa1b5efe10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 354 has memoization hash dbea3445af9489fa97b12f9c027d34de
parsl.dataflow.memoization INFO: Task 354 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16abbba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 119
parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/540b7053-14eb-4abf-8593-64f71ad7acb0_isr_855911_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/540b7053-14eb-4abf-8593-64f71ad7acb0_isr_855911_1.stderr
parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 355 submitted for App characterizeImage, waiting on task 354
parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: <AppFuture at 0x14aa1b5ee810 state=pending>
parsl.dataflow.dflow DEBUG: Task 355 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 356 submitted for App calibrate, waiting on task 355
parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: <AppFuture at 0x14aa1807f550 state=pending>
parsl.dataflow.dflow DEBUG: Task 356 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 357 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: <AppFuture at 0x14aa180f68d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 357 has memoization hash b7bff0fb81f10c26163f2a07aec70fed
parsl.dataflow.memoization INFO: Task 357 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16abbec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 120
parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/222b63fe-32d7-489a-887f-648c77f23dea_isr_855909_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/222b63fe-32d7-489a-887f-648c77f23dea_isr_855909_24.stderr
parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 358 submitted for App characterizeImage, waiting on task 357
parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: <AppFuture at 0x14aa1b5ecd90 state=pending>
parsl.dataflow.dflow DEBUG: Task 358 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 359 submitted for App calibrate, waiting on task 358
parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: <AppFuture at 0x14aa17f8d1d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 359 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 360 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: <AppFuture at 0x14aa17ff3cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 360 has memoization hash de7c4bc6cf97ad80281dd0a36f9a3420
parsl.dataflow.memoization INFO: Task 360 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 121
parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/e859f7bc-52d0-47a5-82f8-3214362b2563_isr_855821_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/e859f7bc-52d0-47a5-82f8-3214362b2563_isr_855821_7.stderr
parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 361 submitted for App characterizeImage, waiting on task 360
parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: <AppFuture at 0x14aa17ff2210 state=pending>
parsl.dataflow.dflow DEBUG: Task 361 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 362 submitted for App calibrate, waiting on task 361
parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: <AppFuture at 0x14aa17ff3690 state=pending>
parsl.dataflow.dflow DEBUG: Task 362 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 363 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: <AppFuture at 0x14aa17e33e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 363 has memoization hash 400fee910008cde85be72ef6c2e0189b
parsl.dataflow.memoization INFO: Task 363 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 122
parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/c6a84b29-e9ae-4f11-beab-1171a7b87217_isr_855911_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/c6a84b29-e9ae-4f11-beab-1171a7b87217_isr_855911_16.stderr
parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 364 submitted for App characterizeImage, waiting on task 363
parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: <AppFuture at 0x14aa17fa3990 state=pending>
parsl.dataflow.dflow DEBUG: Task 364 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 365 submitted for App calibrate, waiting on task 364
parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: <AppFuture at 0x14aa17e7d610 state=pending>
parsl.dataflow.dflow DEBUG: Task 365 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 366 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: <AppFuture at 0x14aa17d7ea50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 366 has memoization hash c360d30e247b60dbd82ab9baced6e43d
parsl.dataflow.memoization INFO: Task 366 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 123
parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/89431413-bc51-4fcb-8471-a763da5c3c5d_isr_855909_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/89431413-bc51-4fcb-8471-a763da5c3c5d_isr_855909_38.stderr
parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 367 submitted for App characterizeImage, waiting on task 366
parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: <AppFuture at 0x14aa173f11d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 367 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 368 submitted for App calibrate, waiting on task 367
parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: <AppFuture at 0x14aa17db31d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 368 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 369 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: <AppFuture at 0x14aa1749df50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 369 has memoization hash 109af9534f35257785143ab312547839
parsl.dataflow.memoization INFO: Task 369 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 124
parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/94573edc-3979-4900-88f5-18ade0306ab8_isr_855911_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/94573edc-3979-4900-88f5-18ade0306ab8_isr_855911_19.stderr
parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 370 submitted for App characterizeImage, waiting on task 369
parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: <AppFuture at 0x14aa1749cc90 state=pending>
parsl.dataflow.dflow DEBUG: Task 370 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 371 submitted for App calibrate, waiting on task 370
parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: <AppFuture at 0x14aa17ca1990 state=pending>
parsl.dataflow.dflow DEBUG: Task 371 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 372 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: <AppFuture at 0x14aa17ca0710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 372 has memoization hash f67634ad5e45bf1e9dfae4927008a446
parsl.dataflow.memoization INFO: Task 372 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 125
parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/fd4b610e-c965-4d35-956f-35b2b9ac3102_isr_855910_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/fd4b610e-c965-4d35-956f-35b2b9ac3102_isr_855910_32.stderr
parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 373 submitted for App characterizeImage, waiting on task 372
parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: <AppFuture at 0x14aa17be32d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 373 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 374 submitted for App calibrate, waiting on task 373
parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: <AppFuture at 0x14aa17adba10 state=pending>
parsl.dataflow.dflow DEBUG: Task 374 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 375 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: <AppFuture at 0x14aa17ad9a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 375 has memoization hash 896cce030c3b70c2e4cf2d95a8f98844
parsl.dataflow.memoization INFO: Task 375 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 126
parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/9e6b6fdb-6aa1-4b7f-8036-5c6e0153dece_isr_855822_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/9e6b6fdb-6aa1-4b7f-8036-5c6e0153dece_isr_855822_42.stderr
parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 376 submitted for App characterizeImage, waiting on task 375
parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: <AppFuture at 0x14aa17adaf90 state=pending>
parsl.dataflow.dflow DEBUG: Task 376 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 377 submitted for App calibrate, waiting on task 376
parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: <AppFuture at 0x14aa19ebb650 state=pending>
parsl.dataflow.dflow DEBUG: Task 377 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 378 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: <AppFuture at 0x14aa1733fa90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 378 has memoization hash 81a0dd1f4a3c31ff9c3aa71c28ad084c
parsl.dataflow.memoization INFO: Task 378 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 127
parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/3b2b70dc-8090-43ed-8ebb-7c2b4c5c737f_isr_855821_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/3b2b70dc-8090-43ed-8ebb-7c2b4c5c737f_isr_855821_48.stderr
parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 379 submitted for App characterizeImage, waiting on task 378
parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: <AppFuture at 0x14aa1733f090 state=pending>
parsl.dataflow.dflow DEBUG: Task 379 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 380 submitted for App calibrate, waiting on task 379
parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: <AppFuture at 0x14aa173a1550 state=pending>
parsl.dataflow.dflow DEBUG: Task 380 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 381 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: <AppFuture at 0x14aa176e98d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 381 has memoization hash 2fe727447528e91b071031de3ec7543d
parsl.dataflow.memoization INFO: Task 381 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 128
parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/a9432ef0-0898-4bb6-822a-1d7b0d9a1d1f_isr_855910_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/a9432ef0-0898-4bb6-822a-1d7b0d9a1d1f_isr_855910_33.stderr
parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 382 submitted for App characterizeImage, waiting on task 381
parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: <AppFuture at 0x14aa1733e9d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 382 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 383 submitted for App calibrate, waiting on task 382
parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: <AppFuture at 0x14aa175c9ad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 383 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 384 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: <AppFuture at 0x14aa1b568890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 384 has memoization hash 9fe2cf8493893579e9e1d260dda8bc3c
parsl.dataflow.memoization INFO: Task 384 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 129
parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/a6f434d5-1d98-4472-92fa-1dba08960f40_isr_855821_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/a6f434d5-1d98-4472-92fa-1dba08960f40_isr_855821_51.stderr
parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 385 submitted for App characterizeImage, waiting on task 384
parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: <AppFuture at 0x14aa18b78890 state=pending>
parsl.dataflow.dflow DEBUG: Task 385 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 386 submitted for App calibrate, waiting on task 385
parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: <AppFuture at 0x14aa1b5699d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 386 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 387 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: <AppFuture at 0x14aa1b56ab50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 387 has memoization hash ad40134f38727f8bb8fd8e3d6a5f5ed7
parsl.dataflow.memoization INFO: Task 387 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 130
parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/b6586e48-4bfd-4638-b7e7-f6519a90d910_isr_855909_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/b6586e48-4bfd-4638-b7e7-f6519a90d910_isr_855909_46.stderr
parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 388 submitted for App characterizeImage, waiting on task 387
parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: <AppFuture at 0x14aa1b56ac90 state=pending>
parsl.dataflow.dflow DEBUG: Task 388 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 389 submitted for App calibrate, waiting on task 388
parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: <AppFuture at 0x14aa1b56bd10 state=pending>
parsl.dataflow.dflow DEBUG: Task 389 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 390 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: <AppFuture at 0x14aa17684f10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 390 has memoization hash c5746101d528f11bf6023f25e4713ad5
parsl.dataflow.memoization INFO: Task 390 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 390 try 0 launched on executor multi with executor id 131
parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/479c12f3-4a9b-4d31-b0c1-9d40260ce965_isr_855822_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/479c12f3-4a9b-4d31-b0c1-9d40260ce965_isr_855822_43.stderr
parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 391 submitted for App characterizeImage, waiting on task 390
parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: <AppFuture at 0x14aa17684510 state=pending>
parsl.dataflow.dflow DEBUG: Task 391 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 392 submitted for App calibrate, waiting on task 391
parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: <AppFuture at 0x14aa175c9e10 state=pending>
parsl.dataflow.dflow DEBUG: Task 392 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 393 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: <AppFuture at 0x14aa173b3d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 393 has memoization hash 14bb492a538bbf169ef54bb7be75df38
parsl.dataflow.memoization INFO: Task 393 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 132
parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/61ab5a2d-d41b-4fc6-bae1-b04e9f849cc7_isr_855822_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/61ab5a2d-d41b-4fc6-bae1-b04e9f849cc7_isr_855822_49.stderr
parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 394 submitted for App characterizeImage, waiting on task 393
parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: <AppFuture at 0x14aa17686010 state=pending>
parsl.dataflow.dflow DEBUG: Task 394 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 395 submitted for App calibrate, waiting on task 394
parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: <AppFuture at 0x14aa172470d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 395 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 396 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: <AppFuture at 0x14aa1b566c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 396 has memoization hash ff4776cbad219c6f8774c535739b6ed7
parsl.dataflow.memoization INFO: Task 396 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 133
parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/baee1d21-bcbf-463d-ba73-9038d1d0e0f9_isr_855820_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/baee1d21-bcbf-463d-ba73-9038d1d0e0f9_isr_855820_45.stderr
parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 397 submitted for App characterizeImage, waiting on task 396
parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: <AppFuture at 0x14aa1b564150 state=pending>
parsl.dataflow.dflow DEBUG: Task 397 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 398 submitted for App calibrate, waiting on task 397
parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: <AppFuture at 0x14aa175247d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 398 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 399 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: <AppFuture at 0x14aa2c3641d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 399 has memoization hash fdca56dfa13bbdd850f8f00d674688f2
parsl.dataflow.memoization INFO: Task 399 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 134
parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/7729a492-8cc1-4f7b-a724-102c7f79271e_isr_855822_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/7729a492-8cc1-4f7b-a724-102c7f79271e_isr_855822_40.stderr
parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 400 submitted for App characterizeImage, waiting on task 399
parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: <AppFuture at 0x14aa2c6e4c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 400 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 401 submitted for App calibrate, waiting on task 400
parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: <AppFuture at 0x14aa1a4e9790 state=pending>
parsl.dataflow.dflow DEBUG: Task 401 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 402 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: <AppFuture at 0x14aa1a4e8650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 402 has memoization hash d7aeac09f50c01205239888d6fb0e924
parsl.dataflow.memoization INFO: Task 402 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 135
parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/56bcc41a-fdb0-4c81-b667-28c0cd9b46fc_isr_855821_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/56bcc41a-fdb0-4c81-b667-28c0cd9b46fc_isr_855821_9.stderr
parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 403 submitted for App characterizeImage, waiting on task 402
parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: <AppFuture at 0x14aa1a4e8310 state=pending>
parsl.dataflow.dflow DEBUG: Task 403 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 404 submitted for App calibrate, waiting on task 403
parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: <AppFuture at 0x14aa1a4fa6d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 404 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 405 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: <AppFuture at 0x14aa1a4fa190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 405 has memoization hash e9960369b5859263af8b9cc7f186acf0
parsl.dataflow.memoization INFO: Task 405 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 136
parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/b6c5e662-93a1-4ebe-8edc-eb1bd66704fe_isr_855911_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/b6c5e662-93a1-4ebe-8edc-eb1bd66704fe_isr_855911_10.stderr
parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 406 submitted for App characterizeImage, waiting on task 405
parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: <AppFuture at 0x14aa1a4fa150 state=pending>
parsl.dataflow.dflow DEBUG: Task 406 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 407 submitted for App calibrate, waiting on task 406
parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: <AppFuture at 0x14aa1a4ebe90 state=pending>
parsl.dataflow.dflow DEBUG: Task 407 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 408 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: <AppFuture at 0x14aa1b709710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 408 has memoization hash 11760383242ae118047b07f7758e1e40
parsl.dataflow.memoization INFO: Task 408 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 137
parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/9dffee86-183f-4d65-a8ac-547374119fd1_isr_855911_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/9dffee86-183f-4d65-a8ac-547374119fd1_isr_855911_42.stderr
parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 409 submitted for App characterizeImage, waiting on task 408
parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: <AppFuture at 0x14aa1b70a610 state=pending>
parsl.dataflow.dflow DEBUG: Task 409 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 410 submitted for App calibrate, waiting on task 409
parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: <AppFuture at 0x14aa1b6ecc50 state=pending>
parsl.dataflow.dflow DEBUG: Task 410 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 411 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: <AppFuture at 0x14aa1b6ed990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 411 has memoization hash 09e0671e2ff9791f9bb962d770341e5f
parsl.dataflow.memoization INFO: Task 411 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 138
parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/721aa592-09ba-43a9-9cd3-8dee11095dc3_isr_855820_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/721aa592-09ba-43a9-9cd3-8dee11095dc3_isr_855820_52.stderr
parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 412 submitted for App characterizeImage, waiting on task 411
parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: <AppFuture at 0x14aa1b359a10 state=pending>
parsl.dataflow.dflow DEBUG: Task 412 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 413 submitted for App calibrate, waiting on task 412
parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: <AppFuture at 0x14aa1b6ec590 state=pending>
parsl.dataflow.dflow DEBUG: Task 413 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 414 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: <AppFuture at 0x14aa1b5f4450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 414 has memoization hash 768708276cf0d74fa130d2a455c5d7d9
parsl.dataflow.memoization INFO: Task 414 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 139
parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/d664ee8f-c140-4718-91e7-c1df9f0aea39_isr_855911_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/d664ee8f-c140-4718-91e7-c1df9f0aea39_isr_855911_7.stderr
parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 415 submitted for App characterizeImage, waiting on task 414
parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: <AppFuture at 0x14aa1b5f7310 state=pending>
parsl.dataflow.dflow DEBUG: Task 415 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 416 submitted for App calibrate, waiting on task 415
parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: <AppFuture at 0x14aa1b5f6990 state=pending>
parsl.dataflow.dflow DEBUG: Task 416 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 417 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: <AppFuture at 0x14aa2c6abb10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 417 has memoization hash 2137c0dfa2d6ed2291c7c92df78fdf31
parsl.dataflow.memoization INFO: Task 417 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 140
parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/ce3a3275-820e-44d0-af01-2f4adb8d848b_isr_855909_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/ce3a3275-820e-44d0-af01-2f4adb8d848b_isr_855909_41.stderr
parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 418 submitted for App characterizeImage, waiting on task 417
parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: <AppFuture at 0x14aa2c6abf90 state=pending>
parsl.dataflow.dflow DEBUG: Task 418 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 419 submitted for App calibrate, waiting on task 418
parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: <AppFuture at 0x14aa2c6aab50 state=pending>
parsl.dataflow.dflow DEBUG: Task 419 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 420 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: <AppFuture at 0x14aa2c6aa250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 420 has memoization hash a2f2b992e21ce3065b93a810807aa89a
parsl.dataflow.memoization INFO: Task 420 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 141
parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/7e1838a3-76d4-4487-9c5b-2ead372bc6ec_isr_855821_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/7e1838a3-76d4-4487-9c5b-2ead372bc6ec_isr_855821_35.stderr
parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 421 submitted for App characterizeImage, waiting on task 420
parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: <AppFuture at 0x14aa2c6aa550 state=pending>
parsl.dataflow.dflow DEBUG: Task 421 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 422 submitted for App calibrate, waiting on task 421
parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: <AppFuture at 0x14aa2c6a8cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 422 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 423 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: <AppFuture at 0x14aa2c6a87d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 423 has memoization hash 929893aee6c7a85a1546f0631ebb1fe2
parsl.dataflow.memoization INFO: Task 423 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 142
parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/6e9990bd-7623-4089-a7c6-0f0b446eaee3_isr_855821_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/6e9990bd-7623-4089-a7c6-0f0b446eaee3_isr_855821_6.stderr
parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 424 submitted for App characterizeImage, waiting on task 423
parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: <AppFuture at 0x14aa2c6a8750 state=pending>
parsl.dataflow.dflow DEBUG: Task 424 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 425 submitted for App calibrate, waiting on task 424
parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: <AppFuture at 0x14aa2c6a8110 state=pending>
parsl.dataflow.dflow DEBUG: Task 425 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 426 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: <AppFuture at 0x14aa2c653510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 426 has memoization hash 902324800f6f75a8dee6941a23dc3368
parsl.dataflow.memoization INFO: Task 426 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 143
parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/1a9b071a-dbbb-46a2-81b2-0a264ad41635_isr_855821_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/1a9b071a-dbbb-46a2-81b2-0a264ad41635_isr_855821_54.stderr
parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 427 submitted for App characterizeImage, waiting on task 426
parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: <AppFuture at 0x14aa2c653310 state=pending>
parsl.dataflow.dflow DEBUG: Task 427 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 428 submitted for App calibrate, waiting on task 427
parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: <AppFuture at 0x14aa2c651f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 428 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 429 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: <AppFuture at 0x14aa2c651a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 429 has memoization hash 6b310fd9f29c2eca3624b35a753ae66a
parsl.dataflow.memoization INFO: Task 429 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 144
parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/a7d6fb5c-17dd-4d51-b8db-70eef770d099_isr_855910_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/a7d6fb5c-17dd-4d51-b8db-70eef770d099_isr_855910_2.stderr
parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 430 submitted for App characterizeImage, waiting on task 429
parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: <AppFuture at 0x14aa2c652a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 430 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 431 submitted for App calibrate, waiting on task 430
parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: <AppFuture at 0x14aa2c6507d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 431 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 432 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: <AppFuture at 0x14aa2c650190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 432 has memoization hash f95964de22098d4d523ada1a8db76d5a
parsl.dataflow.memoization INFO: Task 432 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 145
parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/6ddc11b8-6c3b-446b-b6b4-b193a67a4fd9_isr_855821_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/6ddc11b8-6c3b-446b-b6b4-b193a67a4fd9_isr_855821_20.stderr
parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 433 submitted for App characterizeImage, waiting on task 432
parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: <AppFuture at 0x14aa2c6506d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 433 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 434 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 434 submitted for App calibrate, waiting on task 433
parsl.dataflow.dflow DEBUG: Task 434 set to pending state with AppFuture: <AppFuture at 0x14aa27b778d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 434 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 435 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 435 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 435 set to pending state with AppFuture: <AppFuture at 0x14aa27b76ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 435 has memoization hash f131ac97d2f442e4c8d79d56a63efbcd
parsl.dataflow.memoization INFO: Task 435 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 435 try 0 launched on executor multi with executor id 146
parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/62f4a502-b4e1-4cab-a713-7912b372b5dd_isr_855911_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/62f4a502-b4e1-4cab-a713-7912b372b5dd_isr_855911_60.stderr
parsl.dataflow.dflow DEBUG: Task 436 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 436 submitted for App characterizeImage, waiting on task 435
parsl.dataflow.dflow DEBUG: Task 436 set to pending state with AppFuture: <AppFuture at 0x14aa27b76fd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 436 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 437 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 437 submitted for App calibrate, waiting on task 436
parsl.dataflow.dflow DEBUG: Task 437 set to pending state with AppFuture: <AppFuture at 0x14aa27b76050 state=pending>
parsl.dataflow.dflow DEBUG: Task 437 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 438 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 438 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 438 set to pending state with AppFuture: <AppFuture at 0x14aa27b751d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 438 has memoization hash f77b478bbeb3659b10e76807d732a021
parsl.dataflow.memoization INFO: Task 438 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 438 try 0 launched on executor multi with executor id 147
parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/e5fd3902-9ee5-4857-a222-0775a6053065_isr_855821_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/e5fd3902-9ee5-4857-a222-0775a6053065_isr_855821_24.stderr
parsl.dataflow.dflow DEBUG: Task 439 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 439 submitted for App characterizeImage, waiting on task 438
parsl.dataflow.dflow DEBUG: Task 439 set to pending state with AppFuture: <AppFuture at 0x14aa27b75150 state=pending>
parsl.dataflow.dflow DEBUG: Task 439 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 440 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 440 submitted for App calibrate, waiting on task 439
parsl.dataflow.dflow DEBUG: Task 440 set to pending state with AppFuture: <AppFuture at 0x14aa27b74810 state=pending>
parsl.dataflow.dflow DEBUG: Task 440 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 441 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 441 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 441 set to pending state with AppFuture: <AppFuture at 0x14aa2c6efbd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 441 has memoization hash 81c254272fd08a7e4c883ade2351ae79
parsl.dataflow.memoization INFO: Task 441 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 441 try 0 launched on executor multi with executor id 148
parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/d0804ff0-cb56-44d2-afdc-98bf98c36c99_isr_855910_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/d0804ff0-cb56-44d2-afdc-98bf98c36c99_isr_855910_60.stderr
parsl.dataflow.dflow DEBUG: Task 442 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 442 submitted for App characterizeImage, waiting on task 441
parsl.dataflow.dflow DEBUG: Task 442 set to pending state with AppFuture: <AppFuture at 0x14aa2c6efa50 state=pending>
parsl.dataflow.dflow DEBUG: Task 442 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 443 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 443 submitted for App calibrate, waiting on task 442
parsl.dataflow.dflow DEBUG: Task 443 set to pending state with AppFuture: <AppFuture at 0x14aa2c6ee8d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 443 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 444 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 444 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 444 set to pending state with AppFuture: <AppFuture at 0x14aa2c6ee250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 444 has memoization hash a62fd2a0bc224227173fd3973d81f95b
parsl.dataflow.memoization INFO: Task 444 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 444 try 0 launched on executor multi with executor id 149
parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/6f542525-2f85-48b4-905a-e327cc7555a5_isr_855821_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/6f542525-2f85-48b4-905a-e327cc7555a5_isr_855821_10.stderr
parsl.dataflow.dflow DEBUG: Task 445 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 445 submitted for App characterizeImage, waiting on task 444
parsl.dataflow.dflow DEBUG: Task 445 set to pending state with AppFuture: <AppFuture at 0x14aa2c6ee490 state=pending>
parsl.dataflow.dflow DEBUG: Task 445 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 446 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 446 submitted for App calibrate, waiting on task 445
parsl.dataflow.dflow DEBUG: Task 446 set to pending state with AppFuture: <AppFuture at 0x14aa2c6ecf50 state=pending>
parsl.dataflow.dflow DEBUG: Task 446 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 447 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 447 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 447 set to pending state with AppFuture: <AppFuture at 0x14aa2c6ec890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 447 has memoization hash 9d6d7757848c1b4fc06fbe31dd1cbf53
parsl.dataflow.memoization INFO: Task 447 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 447 try 0 launched on executor multi with executor id 150
parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/6c63572e-47c1-4049-a6ce-dd971070e8fe_isr_855909_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/6c63572e-47c1-4049-a6ce-dd971070e8fe_isr_855909_47.stderr
parsl.dataflow.dflow DEBUG: Task 448 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 448 submitted for App characterizeImage, waiting on task 447
parsl.dataflow.dflow DEBUG: Task 448 set to pending state with AppFuture: <AppFuture at 0x14aa2c6ecc10 state=pending>
parsl.dataflow.dflow DEBUG: Task 448 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 449 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 449 submitted for App calibrate, waiting on task 448
parsl.dataflow.dflow DEBUG: Task 449 set to pending state with AppFuture: <AppFuture at 0x14aa2c6ec210 state=pending>
parsl.dataflow.dflow DEBUG: Task 449 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 450 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 450 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 450 set to pending state with AppFuture: <AppFuture at 0x14aa2c66a3d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 450 has memoization hash 928ca308b039504e531394453ac21932
parsl.dataflow.memoization INFO: Task 450 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 450 try 0 launched on executor multi with executor id 151
parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/87fa736e-3a82-4669-ae46-dcb0bffc97fb_isr_855820_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/87fa736e-3a82-4669-ae46-dcb0bffc97fb_isr_855820_36.stderr
parsl.dataflow.dflow DEBUG: Task 451 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 451 submitted for App characterizeImage, waiting on task 450
parsl.dataflow.dflow DEBUG: Task 451 set to pending state with AppFuture: <AppFuture at 0x14aa2c669c50 state=pending>
parsl.dataflow.dflow DEBUG: Task 451 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 452 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 452 submitted for App calibrate, waiting on task 451
parsl.dataflow.dflow DEBUG: Task 452 set to pending state with AppFuture: <AppFuture at 0x14aa2c66bb90 state=pending>
parsl.dataflow.dflow DEBUG: Task 452 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 453 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 453 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 453 set to pending state with AppFuture: <AppFuture at 0x14aa177089d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 453 has memoization hash e3b4b5994fc76700cb97d17c183cfc54
parsl.dataflow.memoization INFO: Task 453 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 453 try 0 launched on executor multi with executor id 152
parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/0b5a5c15-d743-41e4-8c35-0c8ea255a445_isr_855910_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/0b5a5c15-d743-41e4-8c35-0c8ea255a445_isr_855910_23.stderr
parsl.dataflow.dflow DEBUG: Task 454 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 454 submitted for App characterizeImage, waiting on task 453
parsl.dataflow.dflow DEBUG: Task 454 set to pending state with AppFuture: <AppFuture at 0x14aa2c66b150 state=pending>
parsl.dataflow.dflow DEBUG: Task 454 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 455 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 455 submitted for App calibrate, waiting on task 454
parsl.dataflow.dflow DEBUG: Task 455 set to pending state with AppFuture: <AppFuture at 0x14aa1b5f8810 state=pending>
parsl.dataflow.dflow DEBUG: Task 455 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 456 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 456 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 456 set to pending state with AppFuture: <AppFuture at 0x14aa1a560b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 456 has memoization hash 616dbeefb5551946c298afbaa0ea422f
parsl.dataflow.memoization INFO: Task 456 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 456 try 0 launched on executor multi with executor id 153
parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/376b4276-92bc-402b-8936-5f4b08567a05_isr_855822_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/376b4276-92bc-402b-8936-5f4b08567a05_isr_855822_36.stderr
parsl.dataflow.dflow DEBUG: Task 457 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 457 submitted for App characterizeImage, waiting on task 456
parsl.dataflow.dflow DEBUG: Task 457 set to pending state with AppFuture: <AppFuture at 0x14aa171708d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 457 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 458 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 458 submitted for App calibrate, waiting on task 457
parsl.dataflow.dflow DEBUG: Task 458 set to pending state with AppFuture: <AppFuture at 0x14aa171c6b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 458 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 459 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 459 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 459 set to pending state with AppFuture: <AppFuture at 0x14aa1a4fcf10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 459 has memoization hash d3afe1404b7a10b6b2d2718fc18817cc
parsl.dataflow.memoization INFO: Task 459 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 459 try 0 launched on executor multi with executor id 154
parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/77a69965-45ae-4860-81cb-a88277ccd85d_isr_855820_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/77a69965-45ae-4860-81cb-a88277ccd85d_isr_855820_15.stderr
parsl.dataflow.dflow DEBUG: Task 460 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 460 submitted for App characterizeImage, waiting on task 459
parsl.dataflow.dflow DEBUG: Task 460 set to pending state with AppFuture: <AppFuture at 0x14aa1a4fca90 state=pending>
parsl.dataflow.dflow DEBUG: Task 460 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 461 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 461 submitted for App calibrate, waiting on task 460
parsl.dataflow.dflow DEBUG: Task 461 set to pending state with AppFuture: <AppFuture at 0x14aa1a4fd950 state=pending>
parsl.dataflow.dflow DEBUG: Task 461 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 462 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 462 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 462 set to pending state with AppFuture: <AppFuture at 0x14aa1b563590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 462 has memoization hash 051fe7fae511c820b4ee6cc61700da81
parsl.dataflow.memoization INFO: Task 462 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 462 try 0 launched on executor multi with executor id 155
parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/55ca4f2d-3d61-43ad-be1e-c2eeb94c6600_isr_855911_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/55ca4f2d-3d61-43ad-be1e-c2eeb94c6600_isr_855911_18.stderr
parsl.dataflow.dflow DEBUG: Task 463 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 463 submitted for App characterizeImage, waiting on task 462
parsl.dataflow.dflow DEBUG: Task 463 set to pending state with AppFuture: <AppFuture at 0x14aa1b562590 state=pending>
parsl.dataflow.dflow DEBUG: Task 463 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 464 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 464 submitted for App calibrate, waiting on task 463
parsl.dataflow.dflow DEBUG: Task 464 set to pending state with AppFuture: <AppFuture at 0x14aa1b562390 state=pending>
parsl.dataflow.dflow DEBUG: Task 464 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 465 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 465 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 465 set to pending state with AppFuture: <AppFuture at 0x14aa175b6dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 465 has memoization hash 83c13054b867ff1a540d4caa6e4aa2bd
parsl.dataflow.memoization INFO: Task 465 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 465 try 0 launched on executor multi with executor id 156
parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/590face4-215c-438c-9ee7-d19e59ebc708_isr_855820_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/590face4-215c-438c-9ee7-d19e59ebc708_isr_855820_16.stderr
parsl.dataflow.dflow DEBUG: Task 466 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 466 submitted for App characterizeImage, waiting on task 465
parsl.dataflow.dflow DEBUG: Task 466 set to pending state with AppFuture: <AppFuture at 0x14aa175b6250 state=pending>
parsl.dataflow.dflow DEBUG: Task 466 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 467 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 467 submitted for App calibrate, waiting on task 466
parsl.dataflow.dflow DEBUG: Task 467 set to pending state with AppFuture: <AppFuture at 0x14aa1b562350 state=pending>
parsl.dataflow.dflow DEBUG: Task 467 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 468 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 468 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 468 set to pending state with AppFuture: <AppFuture at 0x14aa182ad9d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 468 has memoization hash a4cbac9c39fa5d13129af4980dd0d0b6
parsl.dataflow.memoization INFO: Task 468 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 468 try 0 launched on executor multi with executor id 157
parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/dda49111-6b68-4dd8-b454-33ef2a3f248e_isr_855822_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/dda49111-6b68-4dd8-b454-33ef2a3f248e_isr_855822_30.stderr
parsl.dataflow.dflow DEBUG: Task 469 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 469 submitted for App characterizeImage, waiting on task 468
parsl.dataflow.dflow DEBUG: Task 469 set to pending state with AppFuture: <AppFuture at 0x14aa182af1d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 469 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 470 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 470 submitted for App calibrate, waiting on task 469
parsl.dataflow.dflow DEBUG: Task 470 set to pending state with AppFuture: <AppFuture at 0x14aa170c5790 state=pending>
parsl.dataflow.dflow DEBUG: Task 470 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 471 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 471 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 471 set to pending state with AppFuture: <AppFuture at 0x14aa17062310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 471 has memoization hash 5d5104c7ea077fd7ef3b799f0886d538
parsl.dataflow.memoization INFO: Task 471 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 471 try 0 launched on executor multi with executor id 158
parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/fd8bf559-d120-4d3f-99bf-46d5e269c954_isr_855820_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/fd8bf559-d120-4d3f-99bf-46d5e269c954_isr_855820_5.stderr
parsl.dataflow.dflow DEBUG: Task 472 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 472 submitted for App characterizeImage, waiting on task 471
parsl.dataflow.dflow DEBUG: Task 472 set to pending state with AppFuture: <AppFuture at 0x14aa17063110 state=pending>
parsl.dataflow.dflow DEBUG: Task 472 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 473 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 473 submitted for App calibrate, waiting on task 472
parsl.dataflow.dflow DEBUG: Task 473 set to pending state with AppFuture: <AppFuture at 0x14aa170724d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 473 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 474 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 474 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 474 set to pending state with AppFuture: <AppFuture at 0x14aa17045990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 474 has memoization hash d41380e772795a90bb355b9e3f340d94
parsl.dataflow.memoization INFO: Task 474 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 474 try 0 launched on executor multi with executor id 159
parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/b140b99f-6507-4d93-bba2-4a363339b6c8_isr_855821_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/b140b99f-6507-4d93-bba2-4a363339b6c8_isr_855821_59.stderr
parsl.dataflow.dflow DEBUG: Task 475 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 475 submitted for App characterizeImage, waiting on task 474
parsl.dataflow.dflow DEBUG: Task 475 set to pending state with AppFuture: <AppFuture at 0x14aa170738d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 475 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 476 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 476 submitted for App calibrate, waiting on task 475
parsl.dataflow.dflow DEBUG: Task 476 set to pending state with AppFuture: <AppFuture at 0x14aa17046910 state=pending>
parsl.dataflow.dflow DEBUG: Task 476 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 477 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 477 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 477 set to pending state with AppFuture: <AppFuture at 0x14aa170d9490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 477 has memoization hash 9d4f8d0114f835545017e0284a39722a
parsl.dataflow.memoization INFO: Task 477 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 477 try 0 launched on executor multi with executor id 160
parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/a25ffa53-f7b0-460a-85c3-9451793e0317_isr_855820_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/a25ffa53-f7b0-460a-85c3-9451793e0317_isr_855820_42.stderr
parsl.dataflow.dflow DEBUG: Task 478 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 478 submitted for App characterizeImage, waiting on task 477
parsl.dataflow.dflow DEBUG: Task 478 set to pending state with AppFuture: <AppFuture at 0x14aa170dbed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 478 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 479 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 479 submitted for App calibrate, waiting on task 478
parsl.dataflow.dflow DEBUG: Task 479 set to pending state with AppFuture: <AppFuture at 0x14aa1701bc10 state=pending>
parsl.dataflow.dflow DEBUG: Task 479 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 480 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 480 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 480 set to pending state with AppFuture: <AppFuture at 0x14aa1701b550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 480 has memoization hash b99aa0d4513c9c4bd467dc2ea54df649
parsl.dataflow.memoization INFO: Task 480 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 480 try 0 launched on executor multi with executor id 161
parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/3f7ed719-c565-4f1e-9134-103224edc290_isr_855909_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/3f7ed719-c565-4f1e-9134-103224edc290_isr_855909_58.stderr
parsl.dataflow.dflow DEBUG: Task 481 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 481 submitted for App characterizeImage, waiting on task 480
parsl.dataflow.dflow DEBUG: Task 481 set to pending state with AppFuture: <AppFuture at 0x14aa17019310 state=pending>
parsl.dataflow.dflow DEBUG: Task 481 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 482 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 482 submitted for App calibrate, waiting on task 481
parsl.dataflow.dflow DEBUG: Task 482 set to pending state with AppFuture: <AppFuture at 0x14aa17100110 state=pending>
parsl.dataflow.dflow DEBUG: Task 482 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 483 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 483 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 483 set to pending state with AppFuture: <AppFuture at 0x14aa17100850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 483 has memoization hash 3c2a78bfc09531b4eb4d2a244ca7469e
parsl.dataflow.memoization INFO: Task 483 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159442c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 483 try 0 launched on executor multi with executor id 162
parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/6b4b6193-8c64-4a36-a782-94d5386be9c2_isr_855820_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/6b4b6193-8c64-4a36-a782-94d5386be9c2_isr_855820_54.stderr
parsl.dataflow.dflow DEBUG: Task 484 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 484 submitted for App characterizeImage, waiting on task 483
parsl.dataflow.dflow DEBUG: Task 484 set to pending state with AppFuture: <AppFuture at 0x14aa17103a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 484 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 485 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 485 submitted for App calibrate, waiting on task 484
parsl.dataflow.dflow DEBUG: Task 485 set to pending state with AppFuture: <AppFuture at 0x14aa17102450 state=pending>
parsl.dataflow.dflow DEBUG: Task 485 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 486 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 486 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 486 set to pending state with AppFuture: <AppFuture at 0x14aa17102a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 486 has memoization hash 970a58ad43b8cde17de85b6d1f947018
parsl.dataflow.memoization INFO: Task 486 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159445e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 486 try 0 launched on executor multi with executor id 163
parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/af8f8a6d-df49-41ec-ac68-181705ad5401_isr_855909_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/af8f8a6d-df49-41ec-ac68-181705ad5401_isr_855909_16.stderr
parsl.dataflow.dflow DEBUG: Task 487 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 487 submitted for App characterizeImage, waiting on task 486
parsl.dataflow.dflow DEBUG: Task 487 set to pending state with AppFuture: <AppFuture at 0x14aa17100950 state=pending>
parsl.dataflow.dflow DEBUG: Task 487 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 488 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 488 submitted for App calibrate, waiting on task 487
parsl.dataflow.dflow DEBUG: Task 488 set to pending state with AppFuture: <AppFuture at 0x14aa170bc690 state=pending>
parsl.dataflow.dflow DEBUG: Task 488 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 489 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 489 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 489 set to pending state with AppFuture: <AppFuture at 0x14aa170bf050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 489 has memoization hash 6e80598e94f615aa6685c4e0677eafee
parsl.dataflow.memoization INFO: Task 489 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15944900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 489 try 0 launched on executor multi with executor id 164
parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/4c8a1b52-ccb1-4c0e-927a-8c1028160320_isr_855821_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/4c8a1b52-ccb1-4c0e-927a-8c1028160320_isr_855821_3.stderr
parsl.dataflow.dflow DEBUG: Task 490 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 490 submitted for App characterizeImage, waiting on task 489
parsl.dataflow.dflow DEBUG: Task 490 set to pending state with AppFuture: <AppFuture at 0x14aa2c6a8090 state=pending>
parsl.dataflow.dflow DEBUG: Task 490 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 491 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 491 submitted for App calibrate, waiting on task 490
parsl.dataflow.dflow DEBUG: Task 491 set to pending state with AppFuture: <AppFuture at 0x14aa170a0550 state=pending>
parsl.dataflow.dflow DEBUG: Task 491 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 492 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 492 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 492 set to pending state with AppFuture: <AppFuture at 0x14aa171f20d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 492 has memoization hash 4261fa4f12168c01072720ac2fb87914
parsl.dataflow.memoization INFO: Task 492 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15944c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 492 try 0 launched on executor multi with executor id 165
parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/3d87f476-30a2-44df-a22d-bbf76d3c3c9e_isr_855911_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/3d87f476-30a2-44df-a22d-bbf76d3c3c9e_isr_855911_43.stderr
parsl.dataflow.dflow DEBUG: Task 493 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 493 submitted for App characterizeImage, waiting on task 492
parsl.dataflow.dflow DEBUG: Task 493 set to pending state with AppFuture: <AppFuture at 0x14aa170a0190 state=pending>
parsl.dataflow.dflow DEBUG: Task 493 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 494 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 494 submitted for App calibrate, waiting on task 493
parsl.dataflow.dflow DEBUG: Task 494 set to pending state with AppFuture: <AppFuture at 0x14aa16f11c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 494 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 495 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 495 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 495 set to pending state with AppFuture: <AppFuture at 0x14aa16f11950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 495 has memoization hash cc583912eebf1c11135d30af840a671e
parsl.dataflow.memoization INFO: Task 495 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15944f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 495 try 0 launched on executor multi with executor id 166
parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/e7d9774b-1689-4e20-8ca0-4c5715afaa73_isr_855909_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/e7d9774b-1689-4e20-8ca0-4c5715afaa73_isr_855909_18.stderr
parsl.dataflow.dflow DEBUG: Task 496 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 496 submitted for App characterizeImage, waiting on task 495
parsl.dataflow.dflow DEBUG: Task 496 set to pending state with AppFuture: <AppFuture at 0x14aa16f10290 state=pending>
parsl.dataflow.dflow DEBUG: Task 496 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 497 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 497 submitted for App calibrate, waiting on task 496
parsl.dataflow.dflow DEBUG: Task 497 set to pending state with AppFuture: <AppFuture at 0x14aa16f12850 state=pending>
parsl.dataflow.dflow DEBUG: Task 497 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 498 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 498 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 498 set to pending state with AppFuture: <AppFuture at 0x14aa16f11a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 498 has memoization hash d249aa45ed05225f299821a449fa1db6
parsl.dataflow.memoization INFO: Task 498 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15945260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 498 try 0 launched on executor multi with executor id 167
parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/9a2fc79b-09e7-4e4e-b4f1-bd6fc7199917_isr_855909_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/9a2fc79b-09e7-4e4e-b4f1-bd6fc7199917_isr_855909_20.stderr
parsl.dataflow.dflow DEBUG: Task 499 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 499 submitted for App characterizeImage, waiting on task 498
parsl.dataflow.dflow DEBUG: Task 499 set to pending state with AppFuture: <AppFuture at 0x14aa16f11d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 499 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 500 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 500 submitted for App calibrate, waiting on task 499
parsl.dataflow.dflow DEBUG: Task 500 set to pending state with AppFuture: <AppFuture at 0x14aa16f265d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 500 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 501 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 501 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 501 set to pending state with AppFuture: <AppFuture at 0x14aa16f26010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 501 has memoization hash f0a745be60295eaac07c0b1a09e6776b
parsl.dataflow.memoization INFO: Task 501 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15945580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 501 try 0 launched on executor multi with executor id 168
parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/58fcaa76-93b1-4654-ac2c-e8a2980d00fa_isr_855820_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/58fcaa76-93b1-4654-ac2c-e8a2980d00fa_isr_855820_62.stderr
parsl.dataflow.dflow DEBUG: Task 502 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 502 submitted for App characterizeImage, waiting on task 501
parsl.dataflow.dflow DEBUG: Task 502 set to pending state with AppFuture: <AppFuture at 0x14aa16f24550 state=pending>
parsl.dataflow.dflow DEBUG: Task 502 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 503 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 503 submitted for App calibrate, waiting on task 502
parsl.dataflow.dflow DEBUG: Task 503 set to pending state with AppFuture: <AppFuture at 0x14aa16f258d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 503 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 504 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 504 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 504 set to pending state with AppFuture: <AppFuture at 0x14aa1708e4d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 504 has memoization hash 1d94b247be3567ec1236e3c7130dc348
parsl.dataflow.memoization INFO: Task 504 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159458a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 504 try 0 launched on executor multi with executor id 169
parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/85ab90c7-ab5c-4a96-a1fc-e8792d265edf_isr_855821_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/85ab90c7-ab5c-4a96-a1fc-e8792d265edf_isr_855821_61.stderr
parsl.dataflow.dflow DEBUG: Task 505 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 505 submitted for App characterizeImage, waiting on task 504
parsl.dataflow.dflow DEBUG: Task 505 set to pending state with AppFuture: <AppFuture at 0x14aa1708f350 state=pending>
parsl.dataflow.dflow DEBUG: Task 505 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 506 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 506 submitted for App calibrate, waiting on task 505
parsl.dataflow.dflow DEBUG: Task 506 set to pending state with AppFuture: <AppFuture at 0x14aa1708e310 state=pending>
parsl.dataflow.dflow DEBUG: Task 506 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 507 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 507 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 507 set to pending state with AppFuture: <AppFuture at 0x14aa16f0ee10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 507 has memoization hash 0240ae59adae360d9068445325303d30
parsl.dataflow.memoization INFO: Task 507 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15945bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 507 try 0 launched on executor multi with executor id 170
parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/f3e8d340-cac8-4010-84ce-817b2694a2f4_isr_855822_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/f3e8d340-cac8-4010-84ce-817b2694a2f4_isr_855822_17.stderr
parsl.dataflow.dflow DEBUG: Task 508 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 508 submitted for App characterizeImage, waiting on task 507
parsl.dataflow.dflow DEBUG: Task 508 set to pending state with AppFuture: <AppFuture at 0x14aa175bd5d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 508 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 509 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 509 submitted for App calibrate, waiting on task 508
parsl.dataflow.dflow DEBUG: Task 509 set to pending state with AppFuture: <AppFuture at 0x14aa16f0d890 state=pending>
parsl.dataflow.dflow DEBUG: Task 509 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 510 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 510 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 510 set to pending state with AppFuture: <AppFuture at 0x14aa16f0d8d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 510 has memoization hash c3ff9f06f8beb6c0adccbb8d9db16a1d
parsl.dataflow.memoization INFO: Task 510 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15945ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 510 try 0 launched on executor multi with executor id 171
parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/e4fb2090-5b57-4d89-865b-d731d42bef35_isr_855911_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/e4fb2090-5b57-4d89-865b-d731d42bef35_isr_855911_28.stderr
parsl.dataflow.dflow DEBUG: Task 511 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 511 submitted for App characterizeImage, waiting on task 510
parsl.dataflow.dflow DEBUG: Task 511 set to pending state with AppFuture: <AppFuture at 0x14aa16f0ca10 state=pending>
parsl.dataflow.dflow DEBUG: Task 511 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 512 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 512 submitted for App calibrate, waiting on task 511
parsl.dataflow.dflow DEBUG: Task 512 set to pending state with AppFuture: <AppFuture at 0x14aa16f0db50 state=pending>
parsl.dataflow.dflow DEBUG: Task 512 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 513 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 513 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 513 set to pending state with AppFuture: <AppFuture at 0x14aa16f0e8d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 513 has memoization hash c1b15705fcad9dae82f090137133e538
parsl.dataflow.memoization INFO: Task 513 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15946200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 513 try 0 launched on executor multi with executor id 172
parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/45c65f4f-700f-408b-9a87-640faf1f8f54_isr_855910_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/45c65f4f-700f-408b-9a87-640faf1f8f54_isr_855910_11.stderr
parsl.dataflow.dflow DEBUG: Task 514 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 514 submitted for App characterizeImage, waiting on task 513
parsl.dataflow.dflow DEBUG: Task 514 set to pending state with AppFuture: <AppFuture at 0x14aa16f0d950 state=pending>
parsl.dataflow.dflow DEBUG: Task 514 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 515 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 515 submitted for App calibrate, waiting on task 514
parsl.dataflow.dflow DEBUG: Task 515 set to pending state with AppFuture: <AppFuture at 0x14aa17093450 state=pending>
parsl.dataflow.dflow DEBUG: Task 515 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 516 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 516 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 516 set to pending state with AppFuture: <AppFuture at 0x14aa17092c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 516 has memoization hash 848f14e1cf13e4e412c786921138bdcb
parsl.dataflow.memoization INFO: Task 516 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15946520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 516 try 0 launched on executor multi with executor id 173
parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/3f1b5134-b85a-412f-8491-80e7d6cfb238_isr_855820_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/3f1b5134-b85a-412f-8491-80e7d6cfb238_isr_855820_38.stderr
parsl.dataflow.dflow DEBUG: Task 517 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 517 submitted for App characterizeImage, waiting on task 516
parsl.dataflow.dflow DEBUG: Task 517 set to pending state with AppFuture: <AppFuture at 0x14aa17090cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 517 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 518 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 518 submitted for App calibrate, waiting on task 517
parsl.dataflow.dflow DEBUG: Task 518 set to pending state with AppFuture: <AppFuture at 0x14aa16f35f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 518 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 519 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 519 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 519 set to pending state with AppFuture: <AppFuture at 0x14aa16f34410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 519 has memoization hash 650dc56c86d26aee674fe5896a85ecbd
parsl.dataflow.memoization INFO: Task 519 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15946840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 519 try 0 launched on executor multi with executor id 174
parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/c82dd4ba-1217-424a-b0bd-1fcf079e013c_isr_855911_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/c82dd4ba-1217-424a-b0bd-1fcf079e013c_isr_855911_11.stderr
parsl.dataflow.dflow DEBUG: Task 520 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 520 submitted for App characterizeImage, waiting on task 519
parsl.dataflow.dflow DEBUG: Task 520 set to pending state with AppFuture: <AppFuture at 0x14aa16f34690 state=pending>
parsl.dataflow.dflow DEBUG: Task 520 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 521 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 521 submitted for App calibrate, waiting on task 520
parsl.dataflow.dflow DEBUG: Task 521 set to pending state with AppFuture: <AppFuture at 0x14aa16f37890 state=pending>
parsl.dataflow.dflow DEBUG: Task 521 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 522 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 522 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 522 set to pending state with AppFuture: <AppFuture at 0x14aa16f37e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 522 has memoization hash 511b6ca10ca87529e78476d38b21cc34
parsl.dataflow.memoization INFO: Task 522 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15946b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 522 try 0 launched on executor multi with executor id 175
parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/88bb420b-e30a-40d2-815f-78922d37dd11_isr_855909_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/88bb420b-e30a-40d2-815f-78922d37dd11_isr_855909_14.stderr
parsl.dataflow.dflow DEBUG: Task 523 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 523 submitted for App characterizeImage, waiting on task 522
parsl.dataflow.dflow DEBUG: Task 523 set to pending state with AppFuture: <AppFuture at 0x14aa16f35390 state=pending>
parsl.dataflow.dflow DEBUG: Task 523 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 524 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 524 submitted for App calibrate, waiting on task 523
parsl.dataflow.dflow DEBUG: Task 524 set to pending state with AppFuture: <AppFuture at 0x14aa15964890 state=pending>
parsl.dataflow.dflow DEBUG: Task 524 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 525 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 525 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 525 set to pending state with AppFuture: <AppFuture at 0x14aa15965350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 525 has memoization hash bcfb52ee0c04ac10dc4cfa2a47bd207e
parsl.dataflow.memoization INFO: Task 525 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15946e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 525 try 0 launched on executor multi with executor id 176
parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/3535da8f-d667-43d7-8994-3f6dc5858503_isr_855911_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/3535da8f-d667-43d7-8994-3f6dc5858503_isr_855911_52.stderr
parsl.dataflow.dflow DEBUG: Task 526 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 526 submitted for App characterizeImage, waiting on task 525
parsl.dataflow.dflow DEBUG: Task 526 set to pending state with AppFuture: <AppFuture at 0x14aa15965d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 526 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 527 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 527 submitted for App calibrate, waiting on task 526
parsl.dataflow.dflow DEBUG: Task 527 set to pending state with AppFuture: <AppFuture at 0x14aa15966010 state=pending>
parsl.dataflow.dflow DEBUG: Task 527 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 528 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 528 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 528 set to pending state with AppFuture: <AppFuture at 0x14aa15966750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 528 has memoization hash da2e349f3017287064d5aeb02308268d
parsl.dataflow.memoization INFO: Task 528 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159471a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 528 try 0 launched on executor multi with executor id 177
parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/6a5be03d-92e4-4514-aaeb-14cf4f806312_isr_855910_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/6a5be03d-92e4-4514-aaeb-14cf4f806312_isr_855910_36.stderr
parsl.dataflow.dflow DEBUG: Task 529 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 529 submitted for App characterizeImage, waiting on task 528
parsl.dataflow.dflow DEBUG: Task 529 set to pending state with AppFuture: <AppFuture at 0x14aa15966890 state=pending>
parsl.dataflow.dflow DEBUG: Task 529 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 530 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 530 submitted for App calibrate, waiting on task 529
parsl.dataflow.dflow DEBUG: Task 530 set to pending state with AppFuture: <AppFuture at 0x14aa15967790 state=pending>
parsl.dataflow.dflow DEBUG: Task 530 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 531 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 531 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 531 set to pending state with AppFuture: <AppFuture at 0x14aa15967ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 531 has memoization hash c54d192225fd5a42d783a6095c627eb8
parsl.dataflow.memoization INFO: Task 531 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159474c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 531 try 0 launched on executor multi with executor id 178
parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/f2974119-4935-4c24-9d9e-870b546d0a76_isr_855910_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/f2974119-4935-4c24-9d9e-870b546d0a76_isr_855910_28.stderr
parsl.dataflow.dflow DEBUG: Task 532 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 532 submitted for App characterizeImage, waiting on task 531
parsl.dataflow.dflow DEBUG: Task 532 set to pending state with AppFuture: <AppFuture at 0x14aa15967ed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 532 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 533 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 533 submitted for App calibrate, waiting on task 532
parsl.dataflow.dflow DEBUG: Task 533 set to pending state with AppFuture: <AppFuture at 0x14aa1596c0d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 533 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 534 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 534 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 534 set to pending state with AppFuture: <AppFuture at 0x14aa1596cf10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 534 has memoization hash 520a2c11f430c7bc446257ab977f9dea
parsl.dataflow.memoization INFO: Task 534 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159477e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 534 try 0 launched on executor multi with executor id 179
parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/55ee9a7d-eceb-4066-b94a-19d15e9ba08e_isr_855822_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/55ee9a7d-eceb-4066-b94a-19d15e9ba08e_isr_855822_27.stderr
parsl.dataflow.dflow DEBUG: Task 535 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 535 submitted for App characterizeImage, waiting on task 534
parsl.dataflow.dflow DEBUG: Task 535 set to pending state with AppFuture: <AppFuture at 0x14aa1596cc50 state=pending>
parsl.dataflow.dflow DEBUG: Task 535 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 536 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 536 submitted for App calibrate, waiting on task 535
parsl.dataflow.dflow DEBUG: Task 536 set to pending state with AppFuture: <AppFuture at 0x14aa1596de50 state=pending>
parsl.dataflow.dflow DEBUG: Task 536 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 537 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 537 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 537 set to pending state with AppFuture: <AppFuture at 0x14aa1596e350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 537 has memoization hash cd29d6d64f6974d867931e39e0fa0967
parsl.dataflow.memoization INFO: Task 537 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15947b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 537 try 0 launched on executor multi with executor id 180
parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/e9decf17-764c-4501-8a27-d6ea7cf65b95_isr_855911_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/e9decf17-764c-4501-8a27-d6ea7cf65b95_isr_855911_51.stderr
parsl.dataflow.dflow DEBUG: Task 538 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 538 submitted for App characterizeImage, waiting on task 537
parsl.dataflow.dflow DEBUG: Task 538 set to pending state with AppFuture: <AppFuture at 0x14aa1596e990 state=pending>
parsl.dataflow.dflow DEBUG: Task 538 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 539 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 539 submitted for App calibrate, waiting on task 538
parsl.dataflow.dflow DEBUG: Task 539 set to pending state with AppFuture: <AppFuture at 0x14aa1596e850 state=pending>
parsl.dataflow.dflow DEBUG: Task 539 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 540 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 540 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 540 set to pending state with AppFuture: <AppFuture at 0x14aa1596f890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 540 has memoization hash 1a9197bf5825eb4d21f3543017180217
parsl.dataflow.memoization INFO: Task 540 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15947e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 540 try 0 launched on executor multi with executor id 181
parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/7f8d73c9-5cff-45ab-b9c5-ef57b5e40676_isr_855911_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/7f8d73c9-5cff-45ab-b9c5-ef57b5e40676_isr_855911_53.stderr
parsl.dataflow.dflow DEBUG: Task 541 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 541 submitted for App characterizeImage, waiting on task 540
parsl.dataflow.dflow DEBUG: Task 541 set to pending state with AppFuture: <AppFuture at 0x14aa1596f490 state=pending>
parsl.dataflow.dflow DEBUG: Task 541 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 542 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 542 submitted for App calibrate, waiting on task 541
parsl.dataflow.dflow DEBUG: Task 542 set to pending state with AppFuture: <AppFuture at 0x14aa15974090 state=pending>
parsl.dataflow.dflow DEBUG: Task 542 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 543 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 543 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 543 set to pending state with AppFuture: <AppFuture at 0x14aa15974c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 543 has memoization hash 455fce76f3a61b396ad90aab73297cf2
parsl.dataflow.memoization INFO: Task 543 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 543 try 0 launched on executor multi with executor id 182
parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/522b43d8-cbbd-4aa7-a1b9-40f1ea9236a7_isr_855822_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/522b43d8-cbbd-4aa7-a1b9-40f1ea9236a7_isr_855822_3.stderr
parsl.dataflow.dflow DEBUG: Task 544 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 544 submitted for App characterizeImage, waiting on task 543
parsl.dataflow.dflow DEBUG: Task 544 set to pending state with AppFuture: <AppFuture at 0x14aa15974310 state=pending>
parsl.dataflow.dflow DEBUG: Task 544 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 545 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 545 submitted for App calibrate, waiting on task 544
parsl.dataflow.dflow DEBUG: Task 545 set to pending state with AppFuture: <AppFuture at 0x14aa15975450 state=pending>
parsl.dataflow.dflow DEBUG: Task 545 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 546 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 546 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 546 set to pending state with AppFuture: <AppFuture at 0x14aa15976050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 546 has memoization hash 176f277f98ecd94da17549b174e455e6
parsl.dataflow.memoization INFO: Task 546 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 546 try 0 launched on executor multi with executor id 183
parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/ad51fae2-0fe9-47b9-b0e3-ed899f41ae08_isr_855910_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/ad51fae2-0fe9-47b9-b0e3-ed899f41ae08_isr_855910_62.stderr
parsl.dataflow.dflow DEBUG: Task 547 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 547 submitted for App characterizeImage, waiting on task 546
parsl.dataflow.dflow DEBUG: Task 547 set to pending state with AppFuture: <AppFuture at 0x14aa15975490 state=pending>
parsl.dataflow.dflow DEBUG: Task 547 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 548 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 548 submitted for App calibrate, waiting on task 547
parsl.dataflow.dflow DEBUG: Task 548 set to pending state with AppFuture: <AppFuture at 0x14aa15976cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 548 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 549 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 549 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 549 set to pending state with AppFuture: <AppFuture at 0x14aa15977410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 549 has memoization hash 237bddaa8110521bb234af855db74521
parsl.dataflow.memoization INFO: Task 549 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 549 try 0 launched on executor multi with executor id 184
parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/ebe913ae-c022-48a1-8738-299200ad6ebb_isr_855821_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/ebe913ae-c022-48a1-8738-299200ad6ebb_isr_855821_44.stderr
parsl.dataflow.dflow DEBUG: Task 550 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 550 submitted for App characterizeImage, waiting on task 549
parsl.dataflow.dflow DEBUG: Task 550 set to pending state with AppFuture: <AppFuture at 0x14aa15977150 state=pending>
parsl.dataflow.dflow DEBUG: Task 550 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 551 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 551 submitted for App calibrate, waiting on task 550
parsl.dataflow.dflow DEBUG: Task 551 set to pending state with AppFuture: <AppFuture at 0x14aa15988110 state=pending>
parsl.dataflow.dflow DEBUG: Task 551 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 552 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 552 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 552 set to pending state with AppFuture: <AppFuture at 0x14aa15988950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 552 has memoization hash fb95d813d3604d41fd91b59ef8638168
parsl.dataflow.memoization INFO: Task 552 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 552 try 0 launched on executor multi with executor id 185
parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/4f5d5307-0b0b-46cb-bb97-e2cfb929cd97_isr_855909_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/4f5d5307-0b0b-46cb-bb97-e2cfb929cd97_isr_855909_48.stderr
parsl.dataflow.dflow DEBUG: Task 553 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 553 submitted for App characterizeImage, waiting on task 552
parsl.dataflow.dflow DEBUG: Task 553 set to pending state with AppFuture: <AppFuture at 0x14aa15977bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 553 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 554 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 554 submitted for App calibrate, waiting on task 553
parsl.dataflow.dflow DEBUG: Task 554 set to pending state with AppFuture: <AppFuture at 0x14aa15989990 state=pending>
parsl.dataflow.dflow DEBUG: Task 554 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 555 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 555 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 555 set to pending state with AppFuture: <AppFuture at 0x14aa15989cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 555 has memoization hash aa40416ea33becf85bddefe70950fb76
parsl.dataflow.memoization INFO: Task 555 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 555 try 0 launched on executor multi with executor id 186
parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/4e8584de-01e9-4b6e-a961-77c618744885_isr_855822_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/4e8584de-01e9-4b6e-a961-77c618744885_isr_855822_35.stderr
parsl.dataflow.dflow DEBUG: Task 556 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 556 submitted for App characterizeImage, waiting on task 555
parsl.dataflow.dflow DEBUG: Task 556 set to pending state with AppFuture: <AppFuture at 0x14aa15989fd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 556 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 557 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 557 submitted for App calibrate, waiting on task 556
parsl.dataflow.dflow DEBUG: Task 557 set to pending state with AppFuture: <AppFuture at 0x14aa1598ad50 state=pending>
parsl.dataflow.dflow DEBUG: Task 557 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 558 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 558 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 558 set to pending state with AppFuture: <AppFuture at 0x14aa1598b250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 558 has memoization hash b100b6f5f65fcbe3c665dd4882a18557
parsl.dataflow.memoization INFO: Task 558 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 558 try 0 launched on executor multi with executor id 187
parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/ebaa991c-af56-4a4e-97a0-9f056406e160_isr_855909_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/ebaa991c-af56-4a4e-97a0-9f056406e160_isr_855909_53.stderr
parsl.dataflow.dflow DEBUG: Task 559 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 559 submitted for App characterizeImage, waiting on task 558
parsl.dataflow.dflow DEBUG: Task 559 set to pending state with AppFuture: <AppFuture at 0x14aa1598b610 state=pending>
parsl.dataflow.dflow DEBUG: Task 559 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 560 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 560 submitted for App calibrate, waiting on task 559
parsl.dataflow.dflow DEBUG: Task 560 set to pending state with AppFuture: <AppFuture at 0x14aa1598b6d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 560 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 561 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 561 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 561 set to pending state with AppFuture: <AppFuture at 0x14aa1598c690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 561 has memoization hash 95f8e9bffdeb25c1f063ab1d7f79fadd
parsl.dataflow.memoization INFO: Task 561 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 561 try 0 launched on executor multi with executor id 188
parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/87caa8a9-bf00-4bb2-8ed6-c603e21ed65a_isr_855821_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/87caa8a9-bf00-4bb2-8ed6-c603e21ed65a_isr_855821_55.stderr
parsl.dataflow.dflow DEBUG: Task 562 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 562 submitted for App characterizeImage, waiting on task 561
parsl.dataflow.dflow DEBUG: Task 562 set to pending state with AppFuture: <AppFuture at 0x14aa1598ce10 state=pending>
parsl.dataflow.dflow DEBUG: Task 562 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 563 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 563 submitted for App calibrate, waiting on task 562
parsl.dataflow.dflow DEBUG: Task 563 set to pending state with AppFuture: <AppFuture at 0x14aa1598cf10 state=pending>
parsl.dataflow.dflow DEBUG: Task 563 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 564 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 564 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 564 set to pending state with AppFuture: <AppFuture at 0x14aa1598da50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 564 has memoization hash 4bfa315d06e36431b1c1ae6c547f8b70
parsl.dataflow.memoization INFO: Task 564 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 564 try 0 launched on executor multi with executor id 189
parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/d8083ccd-bcc1-482c-9eca-fc5f2a94aa78_isr_855822_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/d8083ccd-bcc1-482c-9eca-fc5f2a94aa78_isr_855822_53.stderr
parsl.dataflow.dflow DEBUG: Task 565 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 565 submitted for App characterizeImage, waiting on task 564
parsl.dataflow.dflow DEBUG: Task 565 set to pending state with AppFuture: <AppFuture at 0x14aa1598d110 state=pending>
parsl.dataflow.dflow DEBUG: Task 565 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 566 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 566 submitted for App calibrate, waiting on task 565
parsl.dataflow.dflow DEBUG: Task 566 set to pending state with AppFuture: <AppFuture at 0x14aa1598ea50 state=pending>
parsl.dataflow.dflow DEBUG: Task 566 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 567 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 567 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 567 set to pending state with AppFuture: <AppFuture at 0x14aa1598ee50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 567 has memoization hash 3fb230a6fe0e3c58b4c1cfcdac379ad3
parsl.dataflow.memoization INFO: Task 567 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 567 try 0 launched on executor multi with executor id 190
parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/91c4d1d2-13ac-446e-822a-f87334ee487e_isr_855820_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/91c4d1d2-13ac-446e-822a-f87334ee487e_isr_855820_6.stderr
parsl.dataflow.dflow DEBUG: Task 568 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 568 submitted for App characterizeImage, waiting on task 567
parsl.dataflow.dflow DEBUG: Task 568 set to pending state with AppFuture: <AppFuture at 0x14aa1598cdd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 568 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 569 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 569 submitted for App calibrate, waiting on task 568
parsl.dataflow.dflow DEBUG: Task 569 set to pending state with AppFuture: <AppFuture at 0x14aa1598fed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 569 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 570 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 570 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 570 set to pending state with AppFuture: <AppFuture at 0x14aa1599c050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 570 has memoization hash 23a95ebbe13d259c0bccea1e22ceafe6
parsl.dataflow.memoization INFO: Task 570 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 570 try 0 launched on executor multi with executor id 191
parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/cf0ef916-69b0-48b7-bbee-bc1666a1f629_isr_855911_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/cf0ef916-69b0-48b7-bbee-bc1666a1f629_isr_855911_39.stderr
parsl.dataflow.dflow DEBUG: Task 571 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 571 submitted for App characterizeImage, waiting on task 570
parsl.dataflow.dflow DEBUG: Task 571 set to pending state with AppFuture: <AppFuture at 0x14aa1598fe50 state=pending>
parsl.dataflow.dflow DEBUG: Task 571 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 572 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 572 submitted for App calibrate, waiting on task 571
parsl.dataflow.dflow DEBUG: Task 572 set to pending state with AppFuture: <AppFuture at 0x14aa1599c750 state=pending>
parsl.dataflow.dflow DEBUG: Task 572 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 573 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 573 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 573 set to pending state with AppFuture: <AppFuture at 0x14aa1599d650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 573 has memoization hash 24afe80a1502f0af83c19bd2597d88c7
parsl.dataflow.memoization INFO: Task 573 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 573 try 0 launched on executor multi with executor id 192
parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/ba8e7a7c-9ed0-4453-99b5-d5b28aa51155_isr_855822_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/ba8e7a7c-9ed0-4453-99b5-d5b28aa51155_isr_855822_16.stderr
parsl.dataflow.dflow DEBUG: Task 574 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 574 submitted for App characterizeImage, waiting on task 573
parsl.dataflow.dflow DEBUG: Task 574 set to pending state with AppFuture: <AppFuture at 0x14aa1599dd90 state=pending>
parsl.dataflow.dflow DEBUG: Task 574 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 575 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 575 submitted for App calibrate, waiting on task 574
parsl.dataflow.dflow DEBUG: Task 575 set to pending state with AppFuture: <AppFuture at 0x14aa1599de10 state=pending>
parsl.dataflow.dflow DEBUG: Task 575 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 576 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 576 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 576 set to pending state with AppFuture: <AppFuture at 0x14aa1599ea50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 576 has memoization hash cbf06a97908e6bda8beb3c53339e5f48
parsl.dataflow.memoization INFO: Task 576 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 576 try 0 launched on executor multi with executor id 193
parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/dbeb831e-9585-405f-8eab-4f2bb3bc60f2_isr_855910_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/dbeb831e-9585-405f-8eab-4f2bb3bc60f2_isr_855910_16.stderr
parsl.dataflow.dflow DEBUG: Task 577 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 577 submitted for App characterizeImage, waiting on task 576
parsl.dataflow.dflow DEBUG: Task 577 set to pending state with AppFuture: <AppFuture at 0x14aa1599e790 state=pending>
parsl.dataflow.dflow DEBUG: Task 577 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 578 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 578 submitted for App calibrate, waiting on task 577
parsl.dataflow.dflow DEBUG: Task 578 set to pending state with AppFuture: <AppFuture at 0x14aa1599f4d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 578 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 579 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 579 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 579 set to pending state with AppFuture: <AppFuture at 0x14aa1599fe90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 579 has memoization hash 09f289eb7001b49689d62b354e359dd2
parsl.dataflow.memoization INFO: Task 579 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 579 try 0 launched on executor multi with executor id 194
parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/3689fdee-a1f6-44bc-af7f-b959edfa1e37_isr_855822_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/3689fdee-a1f6-44bc-af7f-b959edfa1e37_isr_855822_51.stderr
parsl.dataflow.dflow DEBUG: Task 580 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 580 submitted for App characterizeImage, waiting on task 579
parsl.dataflow.dflow DEBUG: Task 580 set to pending state with AppFuture: <AppFuture at 0x14aa1599c890 state=pending>
parsl.dataflow.dflow DEBUG: Task 580 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 581 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 581 submitted for App calibrate, waiting on task 580
parsl.dataflow.dflow DEBUG: Task 581 set to pending state with AppFuture: <AppFuture at 0x14aa159a4810 state=pending>
parsl.dataflow.dflow DEBUG: Task 581 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 582 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 582 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 582 set to pending state with AppFuture: <AppFuture at 0x14aa159a5350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 582 has memoization hash 082cc518801e3ed8e60522a01c48bc23
parsl.dataflow.memoization INFO: Task 582 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 582 try 0 launched on executor multi with executor id 195
parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/be1f29ec-6848-43c1-82e8-7f3546e155f3_isr_855820_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/be1f29ec-6848-43c1-82e8-7f3546e155f3_isr_855820_48.stderr
parsl.dataflow.dflow DEBUG: Task 583 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 583 submitted for App characterizeImage, waiting on task 582
parsl.dataflow.dflow DEBUG: Task 583 set to pending state with AppFuture: <AppFuture at 0x14aa159a5d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 583 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 584 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 584 submitted for App calibrate, waiting on task 583
parsl.dataflow.dflow DEBUG: Task 584 set to pending state with AppFuture: <AppFuture at 0x14aa159a5810 state=pending>
parsl.dataflow.dflow DEBUG: Task 584 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 585 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 585 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 585 set to pending state with AppFuture: <AppFuture at 0x14aa159a6810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 585 has memoization hash b5faf1a46519b6f34e545dd418a3b485
parsl.dataflow.memoization INFO: Task 585 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 585 try 0 launched on executor multi with executor id 196
parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/d1a5c6f9-f2a9-49ce-bcff-6308cb46858a_isr_855909_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/d1a5c6f9-f2a9-49ce-bcff-6308cb46858a_isr_855909_50.stderr
parsl.dataflow.dflow DEBUG: Task 586 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 586 submitted for App characterizeImage, waiting on task 585
parsl.dataflow.dflow DEBUG: Task 586 set to pending state with AppFuture: <AppFuture at 0x14aa159a6950 state=pending>
parsl.dataflow.dflow DEBUG: Task 586 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 587 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 587 submitted for App calibrate, waiting on task 586
parsl.dataflow.dflow DEBUG: Task 587 set to pending state with AppFuture: <AppFuture at 0x14aa159a7750 state=pending>
parsl.dataflow.dflow DEBUG: Task 587 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 588 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 588 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 588 set to pending state with AppFuture: <AppFuture at 0x14aa159a7cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 588 has memoization hash eb64482a76e1f4c4e9fbfaa9d9d00b48
parsl.dataflow.memoization INFO: Task 588 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 588 try 0 launched on executor multi with executor id 197
parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/5e383860-1e75-4fd6-ad44-eb03281c056b_isr_855911_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/5e383860-1e75-4fd6-ad44-eb03281c056b_isr_855911_3.stderr
parsl.dataflow.dflow DEBUG: Task 589 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 589 submitted for App characterizeImage, waiting on task 588
parsl.dataflow.dflow DEBUG: Task 589 set to pending state with AppFuture: <AppFuture at 0x14aa159a7990 state=pending>
parsl.dataflow.dflow DEBUG: Task 589 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 590 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 590 submitted for App calibrate, waiting on task 589
parsl.dataflow.dflow DEBUG: Task 590 set to pending state with AppFuture: <AppFuture at 0x14aa159ac490 state=pending>
parsl.dataflow.dflow DEBUG: Task 590 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 591 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 591 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 591 set to pending state with AppFuture: <AppFuture at 0x14aa159ad090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 591 has memoization hash d63f59fe3e1420a5ecfe0ca9297ae462
parsl.dataflow.memoization INFO: Task 591 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 591 try 0 launched on executor multi with executor id 198
parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/f84fab63-a7da-4e22-89ab-8fbea42712e9_isr_855820_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/f84fab63-a7da-4e22-89ab-8fbea42712e9_isr_855820_9.stderr
parsl.dataflow.dflow DEBUG: Task 592 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 592 submitted for App characterizeImage, waiting on task 591
parsl.dataflow.dflow DEBUG: Task 592 set to pending state with AppFuture: <AppFuture at 0x14aa159ad210 state=pending>
parsl.dataflow.dflow DEBUG: Task 592 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 593 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 593 submitted for App calibrate, waiting on task 592
parsl.dataflow.dflow DEBUG: Task 593 set to pending state with AppFuture: <AppFuture at 0x14aa159adf90 state=pending>
parsl.dataflow.dflow DEBUG: Task 593 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 594 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 594 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 594 set to pending state with AppFuture: <AppFuture at 0x14aa159ae490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 594 has memoization hash 34565380a75b92fbe2ef0a2859703d17
parsl.dataflow.memoization INFO: Task 594 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 594 try 0 launched on executor multi with executor id 199
parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/7fed1f33-2b17-48ea-b1ea-bd5421843cc9_isr_855821_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/7fed1f33-2b17-48ea-b1ea-bd5421843cc9_isr_855821_11.stderr
parsl.dataflow.dflow DEBUG: Task 595 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 595 submitted for App characterizeImage, waiting on task 594
parsl.dataflow.dflow DEBUG: Task 595 set to pending state with AppFuture: <AppFuture at 0x14aa159aed10 state=pending>
parsl.dataflow.dflow DEBUG: Task 595 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 596 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 596 submitted for App calibrate, waiting on task 595
parsl.dataflow.dflow DEBUG: Task 596 set to pending state with AppFuture: <AppFuture at 0x14aa159aec90 state=pending>
parsl.dataflow.dflow DEBUG: Task 596 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 597 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 597 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 597 set to pending state with AppFuture: <AppFuture at 0x14aa159af890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 597 has memoization hash 27c113e6ac354ab21056325123795cf5
parsl.dataflow.memoization INFO: Task 597 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 597 try 0 launched on executor multi with executor id 200
parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/73c7702e-3d69-47d2-83b5-efd9ddce1d94_isr_855821_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/73c7702e-3d69-47d2-83b5-efd9ddce1d94_isr_855821_8.stderr
parsl.dataflow.dflow DEBUG: Task 598 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 598 submitted for App characterizeImage, waiting on task 597
parsl.dataflow.dflow DEBUG: Task 598 set to pending state with AppFuture: <AppFuture at 0x14aa159afc10 state=pending>
parsl.dataflow.dflow DEBUG: Task 598 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 599 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 599 submitted for App calibrate, waiting on task 598
parsl.dataflow.dflow DEBUG: Task 599 set to pending state with AppFuture: <AppFuture at 0x14aa159b40d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 599 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 600 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 600 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 600 set to pending state with AppFuture: <AppFuture at 0x14aa159b4c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 600 has memoization hash 234fd90f956d2ebfceb00ce5d3664dad
parsl.dataflow.memoization INFO: Task 600 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 600 try 0 launched on executor multi with executor id 201
parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/a8b197d4-cea4-482a-abf4-e331f426daca_isr_855909_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/a8b197d4-cea4-482a-abf4-e331f426daca_isr_855909_28.stderr
parsl.dataflow.dflow DEBUG: Task 601 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 601 submitted for App characterizeImage, waiting on task 600
parsl.dataflow.dflow DEBUG: Task 601 set to pending state with AppFuture: <AppFuture at 0x14aa159b4110 state=pending>
parsl.dataflow.dflow DEBUG: Task 601 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 602 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 602 submitted for App calibrate, waiting on task 601
parsl.dataflow.dflow DEBUG: Task 602 set to pending state with AppFuture: <AppFuture at 0x14aa159b5950 state=pending>
parsl.dataflow.dflow DEBUG: Task 602 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 603 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 603 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 603 set to pending state with AppFuture: <AppFuture at 0x14aa159b60d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 603 has memoization hash 8fbdf14491c64bef3191fef37dfffd15
parsl.dataflow.memoization INFO: Task 603 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 603 try 0 launched on executor multi with executor id 202
parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/bee74196-931d-4e05-8aae-19bee9c15bee_isr_855822_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/bee74196-931d-4e05-8aae-19bee9c15bee_isr_855822_54.stderr
parsl.dataflow.dflow DEBUG: Task 604 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 604 submitted for App characterizeImage, waiting on task 603
parsl.dataflow.dflow DEBUG: Task 604 set to pending state with AppFuture: <AppFuture at 0x14aa159b5d50 state=pending>
parsl.dataflow.dflow DEBUG: Task 604 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 605 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 605 submitted for App calibrate, waiting on task 604
parsl.dataflow.dflow DEBUG: Task 605 set to pending state with AppFuture: <AppFuture at 0x14aa159b7250 state=pending>
parsl.dataflow.dflow DEBUG: Task 605 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 606 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 606 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 606 set to pending state with AppFuture: <AppFuture at 0x14aa159b7510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 606 has memoization hash f812a861fbb459d25a60924d613eff33
parsl.dataflow.memoization INFO: Task 606 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 606 try 0 launched on executor multi with executor id 203
parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/523f7230-ab5b-40ce-853a-e07c7428fdc3_isr_855822_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/523f7230-ab5b-40ce-853a-e07c7428fdc3_isr_855822_55.stderr
parsl.dataflow.dflow DEBUG: Task 607 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 607 submitted for App characterizeImage, waiting on task 606
parsl.dataflow.dflow DEBUG: Task 607 set to pending state with AppFuture: <AppFuture at 0x14aa159b7d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 607 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 608 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 608 submitted for App calibrate, waiting on task 607
parsl.dataflow.dflow DEBUG: Task 608 set to pending state with AppFuture: <AppFuture at 0x14aa159b7bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 608 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 609 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 609 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 609 set to pending state with AppFuture: <AppFuture at 0x14aa159c4910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 609 has memoization hash 61c3980390d911c44c8cd8ba44f6b69f
parsl.dataflow.memoization INFO: Task 609 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 609 try 0 launched on executor multi with executor id 204
parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/b1d81041-aa2b-48ec-825b-e2898f29bcc2_isr_855820_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/b1d81041-aa2b-48ec-825b-e2898f29bcc2_isr_855820_20.stderr
parsl.dataflow.dflow DEBUG: Task 610 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 610 submitted for App characterizeImage, waiting on task 609
parsl.dataflow.dflow DEBUG: Task 610 set to pending state with AppFuture: <AppFuture at 0x14aa159c4a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 610 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 611 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 611 submitted for App calibrate, waiting on task 610
parsl.dataflow.dflow DEBUG: Task 611 set to pending state with AppFuture: <AppFuture at 0x14aa159c5790 state=pending>
parsl.dataflow.dflow DEBUG: Task 611 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 612 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 612 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 612 set to pending state with AppFuture: <AppFuture at 0x14aa159c5d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 612 has memoization hash 8fc0b462d75ec58ab057117b0a1b131a
parsl.dataflow.memoization INFO: Task 612 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 612 try 0 launched on executor multi with executor id 205
parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/7c078c84-2c06-4ef5-9cc1-dcb0f4e44c89_isr_855910_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/7c078c84-2c06-4ef5-9cc1-dcb0f4e44c89_isr_855910_22.stderr
parsl.dataflow.dflow DEBUG: Task 613 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 613 submitted for App characterizeImage, waiting on task 612
parsl.dataflow.dflow DEBUG: Task 613 set to pending state with AppFuture: <AppFuture at 0x14aa159c5a10 state=pending>
parsl.dataflow.dflow DEBUG: Task 613 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 614 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 614 submitted for App calibrate, waiting on task 613
parsl.dataflow.dflow DEBUG: Task 614 set to pending state with AppFuture: <AppFuture at 0x14aa159c6610 state=pending>
parsl.dataflow.dflow DEBUG: Task 614 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 615 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 615 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 615 set to pending state with AppFuture: <AppFuture at 0x14aa159c7210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 615 has memoization hash f0334225bb989407d3e5c38fe719683a
parsl.dataflow.memoization INFO: Task 615 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 615 try 0 launched on executor multi with executor id 206
parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/13757c90-0ea4-41fa-bbdb-33880cbbfc7b_isr_855911_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/13757c90-0ea4-41fa-bbdb-33880cbbfc7b_isr_855911_17.stderr
parsl.dataflow.dflow DEBUG: Task 616 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 616 submitted for App characterizeImage, waiting on task 615
parsl.dataflow.dflow DEBUG: Task 616 set to pending state with AppFuture: <AppFuture at 0x14aa159c7990 state=pending>
parsl.dataflow.dflow DEBUG: Task 616 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 617 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 617 submitted for App calibrate, waiting on task 616
parsl.dataflow.dflow DEBUG: Task 617 set to pending state with AppFuture: <AppFuture at 0x14aa159c7710 state=pending>
parsl.dataflow.dflow DEBUG: Task 617 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 618 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 618 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 618 set to pending state with AppFuture: <AppFuture at 0x14aa159cc650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 618 has memoization hash ccf968ea855557950f7255fc9141c661
parsl.dataflow.memoization INFO: Task 618 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 618 try 0 launched on executor multi with executor id 207
parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/42f79313-b6fd-4f99-b911-a98977397f7c_isr_855909_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/42f79313-b6fd-4f99-b911-a98977397f7c_isr_855909_54.stderr
parsl.dataflow.dflow DEBUG: Task 619 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 619 submitted for App characterizeImage, waiting on task 618
parsl.dataflow.dflow DEBUG: Task 619 set to pending state with AppFuture: <AppFuture at 0x14aa159cc390 state=pending>
parsl.dataflow.dflow DEBUG: Task 619 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 620 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 620 submitted for App calibrate, waiting on task 619
parsl.dataflow.dflow DEBUG: Task 620 set to pending state with AppFuture: <AppFuture at 0x14aa159ccfd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 620 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 621 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 621 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 621 set to pending state with AppFuture: <AppFuture at 0x14aa159cda10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 621 has memoization hash 6273cd6b33c48e3a942fc95963a6ba20
parsl.dataflow.memoization INFO: Task 621 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 621 try 0 launched on executor multi with executor id 208
parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/1571e2dc-56ec-4200-bd88-e43249792453_isr_855821_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/1571e2dc-56ec-4200-bd88-e43249792453_isr_855821_57.stderr
parsl.dataflow.dflow DEBUG: Task 622 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 622 submitted for App characterizeImage, waiting on task 621
parsl.dataflow.dflow DEBUG: Task 622 set to pending state with AppFuture: <AppFuture at 0x14aa159cd150 state=pending>
parsl.dataflow.dflow DEBUG: Task 622 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 623 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 623 submitted for App calibrate, waiting on task 622
parsl.dataflow.dflow DEBUG: Task 623 set to pending state with AppFuture: <AppFuture at 0x14aa159ce450 state=pending>
parsl.dataflow.dflow DEBUG: Task 623 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 624 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 624 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 624 set to pending state with AppFuture: <AppFuture at 0x14aa159cedd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 624 has memoization hash 1b77bcb0ece42063cb951ced51cd6b9b
parsl.dataflow.memoization INFO: Task 624 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 624 try 0 launched on executor multi with executor id 209
parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/2b38f3e2-f61d-4a3b-b9e5-1f8e2480b279_isr_855820_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/2b38f3e2-f61d-4a3b-b9e5-1f8e2480b279_isr_855820_28.stderr
parsl.dataflow.dflow DEBUG: Task 625 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 625 submitted for App characterizeImage, waiting on task 624
parsl.dataflow.dflow DEBUG: Task 625 set to pending state with AppFuture: <AppFuture at 0x14aa159ceb10 state=pending>
parsl.dataflow.dflow DEBUG: Task 625 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 626 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 626 submitted for App calibrate, waiting on task 625
parsl.dataflow.dflow DEBUG: Task 626 set to pending state with AppFuture: <AppFuture at 0x14aa159cf710 state=pending>
parsl.dataflow.dflow DEBUG: Task 626 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 627 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 627 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 627 set to pending state with AppFuture: <AppFuture at 0x14aa157e0050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 627 has memoization hash 00e33f9331a09d44ce5ffa9c875fff5d
parsl.dataflow.memoization INFO: Task 627 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 627 try 0 launched on executor multi with executor id 210
parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/dee9523a-cb65-419c-88d5-cf697a914f4a_isr_855822_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/dee9523a-cb65-419c-88d5-cf697a914f4a_isr_855822_7.stderr
parsl.dataflow.dflow DEBUG: Task 628 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 628 submitted for App characterizeImage, waiting on task 627
parsl.dataflow.dflow DEBUG: Task 628 set to pending state with AppFuture: <AppFuture at 0x14aa159cfe50 state=pending>
parsl.dataflow.dflow DEBUG: Task 628 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 629 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 629 submitted for App calibrate, waiting on task 628
parsl.dataflow.dflow DEBUG: Task 629 set to pending state with AppFuture: <AppFuture at 0x14aa157e1150 state=pending>
parsl.dataflow.dflow DEBUG: Task 629 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 630 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 630 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 630 set to pending state with AppFuture: <AppFuture at 0x14aa157e1650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 630 has memoization hash 4fa415be8fd027788e3f1a353aa5981c
parsl.dataflow.memoization INFO: Task 630 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 630 try 0 launched on executor multi with executor id 211
parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/bd081662-815f-4abf-8863-24827cf0963e_isr_855820_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/bd081662-815f-4abf-8863-24827cf0963e_isr_855820_8.stderr
parsl.dataflow.dflow DEBUG: Task 631 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 631 submitted for App characterizeImage, waiting on task 630
parsl.dataflow.dflow DEBUG: Task 631 set to pending state with AppFuture: <AppFuture at 0x14aa157e2590 state=pending>
parsl.dataflow.dflow DEBUG: Task 631 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 632 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 632 submitted for App calibrate, waiting on task 631
parsl.dataflow.dflow DEBUG: Task 632 set to pending state with AppFuture: <AppFuture at 0x14aa157e1b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 632 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 633 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 633 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 633 set to pending state with AppFuture: <AppFuture at 0x14aa157e2b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 633 has memoization hash 0d629e3ebd5d589e0cd2bad1745104d2
parsl.dataflow.memoization INFO: Task 633 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 633 try 0 launched on executor multi with executor id 212
parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/daee3196-f336-4169-8bd6-4ac69dbcd0ea_isr_855820_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/daee3196-f336-4169-8bd6-4ac69dbcd0ea_isr_855820_19.stderr
parsl.dataflow.dflow DEBUG: Task 634 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 634 submitted for App characterizeImage, waiting on task 633
parsl.dataflow.dflow DEBUG: Task 634 set to pending state with AppFuture: <AppFuture at 0x14aa157e3490 state=pending>
parsl.dataflow.dflow DEBUG: Task 634 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 635 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 635 submitted for App calibrate, waiting on task 634
parsl.dataflow.dflow DEBUG: Task 635 set to pending state with AppFuture: <AppFuture at 0x14aa157e3bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 635 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 636 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 636 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 636 set to pending state with AppFuture: <AppFuture at 0x14aa157f0110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 636 has memoization hash f82b7e423647e0e79b3c452353fcdb31
parsl.dataflow.memoization INFO: Task 636 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 636 try 0 launched on executor multi with executor id 213
parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/1e0e4c2a-bf53-449f-ad6e-953b69e6b76d_isr_855911_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/1e0e4c2a-bf53-449f-ad6e-953b69e6b76d_isr_855911_49.stderr
parsl.dataflow.dflow DEBUG: Task 637 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 637 submitted for App characterizeImage, waiting on task 636
parsl.dataflow.dflow DEBUG: Task 637 set to pending state with AppFuture: <AppFuture at 0x14aa157e3d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 637 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 638 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 638 submitted for App calibrate, waiting on task 637
parsl.dataflow.dflow DEBUG: Task 638 set to pending state with AppFuture: <AppFuture at 0x14aa157f07d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 638 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 639 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 639 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 639 set to pending state with AppFuture: <AppFuture at 0x14aa157f1450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 639 has memoization hash bfd7cb72c92feddf9d64ca642ae6cd9f
parsl.dataflow.memoization INFO: Task 639 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 639 try 0 launched on executor multi with executor id 214
parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/af248e94-7a3c-45d1-9f37-8e9bf35355c5_isr_855910_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/af248e94-7a3c-45d1-9f37-8e9bf35355c5_isr_855910_25.stderr
parsl.dataflow.dflow DEBUG: Task 640 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 640 submitted for App characterizeImage, waiting on task 639
parsl.dataflow.dflow DEBUG: Task 640 set to pending state with AppFuture: <AppFuture at 0x14aa157e3890 state=pending>
parsl.dataflow.dflow DEBUG: Task 640 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 641 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 641 submitted for App calibrate, waiting on task 640
parsl.dataflow.dflow DEBUG: Task 641 set to pending state with AppFuture: <AppFuture at 0x14aa157f1950 state=pending>
parsl.dataflow.dflow DEBUG: Task 641 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 642 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 642 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 642 set to pending state with AppFuture: <AppFuture at 0x14aa157f2890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 642 has memoization hash 2d7000ff65c56ae5e46bd130b33a9462
parsl.dataflow.memoization INFO: Task 642 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 642 try 0 launched on executor multi with executor id 215
parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/07bd7b60-05ee-4d79-8346-43e0b2a830b0_isr_855822_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/07bd7b60-05ee-4d79-8346-43e0b2a830b0_isr_855822_45.stderr
parsl.dataflow.dflow DEBUG: Task 643 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 643 submitted for App characterizeImage, waiting on task 642
parsl.dataflow.dflow DEBUG: Task 643 set to pending state with AppFuture: <AppFuture at 0x14aa157f1c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 643 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 644 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 644 submitted for App calibrate, waiting on task 643
parsl.dataflow.dflow DEBUG: Task 644 set to pending state with AppFuture: <AppFuture at 0x14aa157f35d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 644 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 645 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 645 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 645 set to pending state with AppFuture: <AppFuture at 0x14aa157f3c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 645 has memoization hash e1e201db567655dfcdfad39fb3c17118
parsl.dataflow.memoization INFO: Task 645 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 645 try 0 launched on executor multi with executor id 216
parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/91cbe4b7-1b4d-4e21-a9d3-6a70c3ebe35e_isr_855909_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/91cbe4b7-1b4d-4e21-a9d3-6a70c3ebe35e_isr_855909_56.stderr
parsl.dataflow.dflow DEBUG: Task 646 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 646 submitted for App characterizeImage, waiting on task 645
parsl.dataflow.dflow DEBUG: Task 646 set to pending state with AppFuture: <AppFuture at 0x14aa157f39d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 646 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 647 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 647 submitted for App calibrate, waiting on task 646
parsl.dataflow.dflow DEBUG: Task 647 set to pending state with AppFuture: <AppFuture at 0x14aa157f4590 state=pending>
parsl.dataflow.dflow DEBUG: Task 647 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 648 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 648 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 648 set to pending state with AppFuture: <AppFuture at 0x14aa157f50d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 648 has memoization hash 91494feb39ecd9b5ea8144b906acda99
parsl.dataflow.memoization INFO: Task 648 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 648 try 0 launched on executor multi with executor id 217
parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/99ed01cf-9567-4a33-b13e-83ffccfd02dc_isr_855911_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/99ed01cf-9567-4a33-b13e-83ffccfd02dc_isr_855911_12.stderr
parsl.dataflow.dflow DEBUG: Task 649 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 649 submitted for App characterizeImage, waiting on task 648
parsl.dataflow.dflow DEBUG: Task 649 set to pending state with AppFuture: <AppFuture at 0x14aa157f5210 state=pending>
parsl.dataflow.dflow DEBUG: Task 649 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 650 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 650 submitted for App calibrate, waiting on task 649
parsl.dataflow.dflow DEBUG: Task 650 set to pending state with AppFuture: <AppFuture at 0x14aa157f5f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 650 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 651 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 651 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 651 set to pending state with AppFuture: <AppFuture at 0x14aa157f6550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 651 has memoization hash c640c2c85c43eb654db99b71e4f34cc8
parsl.dataflow.memoization INFO: Task 651 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 651 try 0 launched on executor multi with executor id 218
parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/d13986ba-7ea0-4590-b5e2-8ca97fc487f5_isr_855820_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/d13986ba-7ea0-4590-b5e2-8ca97fc487f5_isr_855820_61.stderr
parsl.dataflow.dflow DEBUG: Task 652 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 652 submitted for App characterizeImage, waiting on task 651
parsl.dataflow.dflow DEBUG: Task 652 set to pending state with AppFuture: <AppFuture at 0x14aa157f66d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 652 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 653 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 653 submitted for App calibrate, waiting on task 652
parsl.dataflow.dflow DEBUG: Task 653 set to pending state with AppFuture: <AppFuture at 0x14aa157f72d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 653 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 654 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 654 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 654 set to pending state with AppFuture: <AppFuture at 0x14aa157f7a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 654 has memoization hash 07b2f740b6dc567d0c93ee787298b93a
parsl.dataflow.memoization INFO: Task 654 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 654 try 0 launched on executor multi with executor id 219
parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/9df2a49f-3bb9-400b-9b45-45603ca2b42c_isr_855822_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/9df2a49f-3bb9-400b-9b45-45603ca2b42c_isr_855822_37.stderr
parsl.dataflow.dflow DEBUG: Task 655 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 655 submitted for App characterizeImage, waiting on task 654
parsl.dataflow.dflow DEBUG: Task 655 set to pending state with AppFuture: <AppFuture at 0x14aa157f7dd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 655 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 656 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 656 submitted for App calibrate, waiting on task 655
parsl.dataflow.dflow DEBUG: Task 656 set to pending state with AppFuture: <AppFuture at 0x14aa158008d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 656 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 657 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 657 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 657 set to pending state with AppFuture: <AppFuture at 0x14aa15800e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 657 has memoization hash dde366b1c2381119f7745fcf191df056
parsl.dataflow.memoization INFO: Task 657 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 657 try 0 launched on executor multi with executor id 220
parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/66915e4b-d5b6-4d47-94b5-238f5c5f425b_isr_855909_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/66915e4b-d5b6-4d47-94b5-238f5c5f425b_isr_855909_52.stderr
parsl.dataflow.dflow DEBUG: Task 658 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 658 submitted for App characterizeImage, waiting on task 657
parsl.dataflow.dflow DEBUG: Task 658 set to pending state with AppFuture: <AppFuture at 0x14aa15800290 state=pending>
parsl.dataflow.dflow DEBUG: Task 658 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 659 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 659 submitted for App calibrate, waiting on task 658
parsl.dataflow.dflow DEBUG: Task 659 set to pending state with AppFuture: <AppFuture at 0x14aa15801650 state=pending>
parsl.dataflow.dflow DEBUG: Task 659 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 660 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 660 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 660 set to pending state with AppFuture: <AppFuture at 0x14aa158021d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 660 has memoization hash f09671bf22f6ff046a6e6af48b3d6254
parsl.dataflow.memoization INFO: Task 660 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 660 try 0 launched on executor multi with executor id 221
parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/778d4852-0f3b-4d0e-b09e-a2ebdc546423_isr_855820_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/778d4852-0f3b-4d0e-b09e-a2ebdc546423_isr_855820_44.stderr
parsl.dataflow.dflow DEBUG: Task 661 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 661 submitted for App characterizeImage, waiting on task 660
parsl.dataflow.dflow DEBUG: Task 661 set to pending state with AppFuture: <AppFuture at 0x14aa15801f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 661 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 662 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 662 submitted for App calibrate, waiting on task 661
parsl.dataflow.dflow DEBUG: Task 662 set to pending state with AppFuture: <AppFuture at 0x14aa15802a90 state=pending>
parsl.dataflow.dflow DEBUG: Task 662 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 663 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 663 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 663 set to pending state with AppFuture: <AppFuture at 0x14aa15803610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 663 has memoization hash b81be6681b4e7ba740e90241f1f7062d
parsl.dataflow.memoization INFO: Task 663 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 663 try 0 launched on executor multi with executor id 222
parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/b91004a8-4e03-49c9-bdc4-405cb3cdbe3b_isr_855910_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/b91004a8-4e03-49c9-bdc4-405cb3cdbe3b_isr_855910_46.stderr
parsl.dataflow.dflow DEBUG: Task 664 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 664 submitted for App characterizeImage, waiting on task 663
parsl.dataflow.dflow DEBUG: Task 664 set to pending state with AppFuture: <AppFuture at 0x14aa15803910 state=pending>
parsl.dataflow.dflow DEBUG: Task 664 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 665 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 665 submitted for App calibrate, waiting on task 664
parsl.dataflow.dflow DEBUG: Task 665 set to pending state with AppFuture: <AppFuture at 0x14aa1580c510 state=pending>
parsl.dataflow.dflow DEBUG: Task 665 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 666 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 666 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 666 set to pending state with AppFuture: <AppFuture at 0x14aa1580ca50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 666 has memoization hash 7b997c6a08b622bf86b2e9432594ce71
parsl.dataflow.memoization INFO: Task 666 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15808220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 666 try 0 launched on executor multi with executor id 223
parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/16898732-a5dc-41d0-bcf0-15a9f84a964f_isr_855909_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/16898732-a5dc-41d0-bcf0-15a9f84a964f_isr_855909_33.stderr
parsl.dataflow.dflow DEBUG: Task 667 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 667 submitted for App characterizeImage, waiting on task 666
parsl.dataflow.dflow DEBUG: Task 667 set to pending state with AppFuture: <AppFuture at 0x14aa1596d550 state=pending>
parsl.dataflow.dflow DEBUG: Task 667 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 668 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 668 submitted for App calibrate, waiting on task 667
parsl.dataflow.dflow DEBUG: Task 668 set to pending state with AppFuture: <AppFuture at 0x14aa1580cf50 state=pending>
parsl.dataflow.dflow DEBUG: Task 668 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 669 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 669 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 669 set to pending state with AppFuture: <AppFuture at 0x14aa1580de50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 669 has memoization hash 3b499dcc083f40615155d5b1d440a7f1
parsl.dataflow.memoization INFO: Task 669 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15808540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 669 try 0 launched on executor multi with executor id 224
parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/ed99eb83-f804-4f46-af1e-5c83c40aba8c_isr_855910_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/ed99eb83-f804-4f46-af1e-5c83c40aba8c_isr_855910_19.stderr
parsl.dataflow.dflow DEBUG: Task 670 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 670 submitted for App characterizeImage, waiting on task 669
parsl.dataflow.dflow DEBUG: Task 670 set to pending state with AppFuture: <AppFuture at 0x14aa1580d350 state=pending>
parsl.dataflow.dflow DEBUG: Task 670 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 671 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 671 submitted for App calibrate, waiting on task 670
parsl.dataflow.dflow DEBUG: Task 671 set to pending state with AppFuture: <AppFuture at 0x14aa1580ead0 state=pending>
parsl.dataflow.dflow DEBUG: Task 671 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 672 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 672 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 672 set to pending state with AppFuture: <AppFuture at 0x14aa1580f290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 672 has memoization hash 872d0e32e983cc9b8b489722c033880c
parsl.dataflow.memoization INFO: Task 672 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15808860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 672 try 0 launched on executor multi with executor id 225
parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/487c3ba3-82a2-4164-9564-83657118b9c6_isr_855821_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/487c3ba3-82a2-4164-9564-83657118b9c6_isr_855821_49.stderr
parsl.dataflow.dflow DEBUG: Task 673 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 673 submitted for App characterizeImage, waiting on task 672
parsl.dataflow.dflow DEBUG: Task 673 set to pending state with AppFuture: <AppFuture at 0x14aa1580f610 state=pending>
parsl.dataflow.dflow DEBUG: Task 673 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 674 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 674 submitted for App calibrate, waiting on task 673
parsl.dataflow.dflow DEBUG: Task 674 set to pending state with AppFuture: <AppFuture at 0x14aa1580f750 state=pending>
parsl.dataflow.dflow DEBUG: Task 674 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 675 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 675 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 675 set to pending state with AppFuture: <AppFuture at 0x14aa15818690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 675 has memoization hash c24ccbeb3e15ca90db20615ab390a6c0
parsl.dataflow.memoization INFO: Task 675 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15808b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 675 try 0 launched on executor multi with executor id 226
parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/935eb0ec-9d94-489a-aa44-b15e717c0003_isr_855911_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/935eb0ec-9d94-489a-aa44-b15e717c0003_isr_855911_62.stderr
parsl.dataflow.dflow DEBUG: Task 676 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 676 submitted for App characterizeImage, waiting on task 675
parsl.dataflow.dflow DEBUG: Task 676 set to pending state with AppFuture: <AppFuture at 0x14aa15988f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 676 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 677 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 677 submitted for App calibrate, waiting on task 676
parsl.dataflow.dflow DEBUG: Task 677 set to pending state with AppFuture: <AppFuture at 0x14aa158190d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 677 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 678 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 678 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 678 set to pending state with AppFuture: <AppFuture at 0x14aa15819ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 678 has memoization hash 810b004fa849cacf2639f85473528862
parsl.dataflow.memoization INFO: Task 678 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15808ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 678 try 0 launched on executor multi with executor id 227
parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/3ed9417e-f2eb-4b0f-aae6-3a810f150c84_isr_855911_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/3ed9417e-f2eb-4b0f-aae6-3a810f150c84_isr_855911_54.stderr
parsl.dataflow.dflow DEBUG: Task 679 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 679 submitted for App characterizeImage, waiting on task 678
parsl.dataflow.dflow DEBUG: Task 679 set to pending state with AppFuture: <AppFuture at 0x14aa1581a4d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 679 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 680 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 680 submitted for App calibrate, waiting on task 679
parsl.dataflow.dflow DEBUG: Task 680 set to pending state with AppFuture: <AppFuture at 0x14aa1581a3d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 680 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 681 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 681 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 681 set to pending state with AppFuture: <AppFuture at 0x14aa1581afd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 681 has memoization hash f9af664cfebd608ad192b5e6ec03748a
parsl.dataflow.memoization INFO: Task 681 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158091c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 681 try 0 launched on executor multi with executor id 228
parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/4f6ccac5-ea84-4018-b9b1-9821a726d447_isr_855911_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/4f6ccac5-ea84-4018-b9b1-9821a726d447_isr_855911_25.stderr
parsl.dataflow.dflow DEBUG: Task 682 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 682 submitted for App characterizeImage, waiting on task 681
parsl.dataflow.dflow DEBUG: Task 682 set to pending state with AppFuture: <AppFuture at 0x14aa1581a9d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 682 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 683 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 683 submitted for App calibrate, waiting on task 682
parsl.dataflow.dflow DEBUG: Task 683 set to pending state with AppFuture: <AppFuture at 0x14aa1581b810 state=pending>
parsl.dataflow.dflow DEBUG: Task 683 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 684 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 684 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 684 set to pending state with AppFuture: <AppFuture at 0x14aa15820390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 684 has memoization hash dbef9fc5591fc757f468c6d0de0c5a7e
parsl.dataflow.memoization INFO: Task 684 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158094e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 684 try 0 launched on executor multi with executor id 229
parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/f4a97481-681f-4dbd-b41a-9d278275fef9_isr_855820_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/f4a97481-681f-4dbd-b41a-9d278275fef9_isr_855820_7.stderr
parsl.dataflow.dflow DEBUG: Task 685 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 685 submitted for App characterizeImage, waiting on task 684
parsl.dataflow.dflow DEBUG: Task 685 set to pending state with AppFuture: <AppFuture at 0x14aa15820510 state=pending>
parsl.dataflow.dflow DEBUG: Task 685 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 686 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 686 submitted for App calibrate, waiting on task 685
parsl.dataflow.dflow DEBUG: Task 686 set to pending state with AppFuture: <AppFuture at 0x14aa158211d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 686 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 687 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 687 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 687 set to pending state with AppFuture: <AppFuture at 0x14aa15821750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 687 has memoization hash 1c6ea178e21a6b8bea4eb92f04eb83cf
parsl.dataflow.memoization INFO: Task 687 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15809800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 687 try 0 launched on executor multi with executor id 230
parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/328c9de3-5408-4ce7-83af-9ec14f4a25de_isr_855821_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/328c9de3-5408-4ce7-83af-9ec14f4a25de_isr_855821_41.stderr
parsl.dataflow.dflow DEBUG: Task 688 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 688 submitted for App characterizeImage, waiting on task 687
parsl.dataflow.dflow DEBUG: Task 688 set to pending state with AppFuture: <AppFuture at 0x14aa15820dd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 688 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 689 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 689 submitted for App calibrate, waiting on task 688
parsl.dataflow.dflow DEBUG: Task 689 set to pending state with AppFuture: <AppFuture at 0x14aa15822010 state=pending>
parsl.dataflow.dflow DEBUG: Task 689 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 690 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 690 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 690 set to pending state with AppFuture: <AppFuture at 0x14aa15822c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 690 has memoization hash b9a649f466c4a2b51bb3988d9790a7aa
parsl.dataflow.memoization INFO: Task 690 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15809b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 690 try 0 launched on executor multi with executor id 231
parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/dd73294b-184a-4946-b93b-be3fdc1a32bc_isr_855822_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/dd73294b-184a-4946-b93b-be3fdc1a32bc_isr_855822_50.stderr
parsl.dataflow.dflow DEBUG: Task 691 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 691 submitted for App characterizeImage, waiting on task 690
parsl.dataflow.dflow DEBUG: Task 691 set to pending state with AppFuture: <AppFuture at 0x14aa15821f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 691 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 692 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 692 submitted for App calibrate, waiting on task 691
parsl.dataflow.dflow DEBUG: Task 692 set to pending state with AppFuture: <AppFuture at 0x14aa158234d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 692 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 693 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 693 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 693 set to pending state with AppFuture: <AppFuture at 0x14aa15823f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 693 has memoization hash 4e71f9517634b9cd001147ebcda1ef0b
parsl.dataflow.memoization INFO: Task 693 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15809e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 693 try 0 launched on executor multi with executor id 232
parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/31152fe2-2e7d-46ad-8a97-1eecd170a7b4_isr_855911_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/31152fe2-2e7d-46ad-8a97-1eecd170a7b4_isr_855911_44.stderr
parsl.dataflow.dflow DEBUG: Task 694 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 694 submitted for App characterizeImage, waiting on task 693
parsl.dataflow.dflow DEBUG: Task 694 set to pending state with AppFuture: <AppFuture at 0x14aa15823cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 694 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 695 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 695 submitted for App calibrate, waiting on task 694
parsl.dataflow.dflow DEBUG: Task 695 set to pending state with AppFuture: <AppFuture at 0x14aa1582d090 state=pending>
parsl.dataflow.dflow DEBUG: Task 695 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 696 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 696 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 696 set to pending state with AppFuture: <AppFuture at 0x14aa1582d3d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 696 has memoization hash 9065b6305d94d4a456ef2e0df3255936
parsl.dataflow.memoization INFO: Task 696 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 696 try 0 launched on executor multi with executor id 233
parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/7f7f12fa-f45f-4ed6-942c-30a21b76465a_isr_855910_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/7f7f12fa-f45f-4ed6-942c-30a21b76465a_isr_855910_21.stderr
parsl.dataflow.dflow DEBUG: Task 697 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 697 submitted for App characterizeImage, waiting on task 696
parsl.dataflow.dflow DEBUG: Task 697 set to pending state with AppFuture: <AppFuture at 0x14aa1582d0d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 697 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 698 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 698 submitted for App calibrate, waiting on task 697
parsl.dataflow.dflow DEBUG: Task 698 set to pending state with AppFuture: <AppFuture at 0x14aa1582dc50 state=pending>
parsl.dataflow.dflow DEBUG: Task 698 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 699 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 699 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 699 set to pending state with AppFuture: <AppFuture at 0x14aa1582e810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 699 has memoization hash ca8395580f5d05902d6ca1b402b82dd7
parsl.dataflow.memoization INFO: Task 699 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 699 try 0 launched on executor multi with executor id 234
parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/af2b664c-9f37-4363-aca5-6ee11507e9e9_isr_855821_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/af2b664c-9f37-4363-aca5-6ee11507e9e9_isr_855821_45.stderr
parsl.dataflow.dflow DEBUG: Task 700 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 700 submitted for App characterizeImage, waiting on task 699
parsl.dataflow.dflow DEBUG: Task 700 set to pending state with AppFuture: <AppFuture at 0x14aa1582de50 state=pending>
parsl.dataflow.dflow DEBUG: Task 700 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 701 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 701 submitted for App calibrate, waiting on task 700
parsl.dataflow.dflow DEBUG: Task 701 set to pending state with AppFuture: <AppFuture at 0x14aa1582f110 state=pending>
parsl.dataflow.dflow DEBUG: Task 701 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 702 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 702 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 702 set to pending state with AppFuture: <AppFuture at 0x14aa1582fc10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 702 has memoization hash 7e6b9442e0e8255d9e5ef5a3a8ed3328
parsl.dataflow.memoization INFO: Task 702 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 702 try 0 launched on executor multi with executor id 235
parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/fc43ee7e-dd81-491c-8b4f-5456638e7c0f_isr_855909_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/fc43ee7e-dd81-491c-8b4f-5456638e7c0f_isr_855909_27.stderr
parsl.dataflow.dflow DEBUG: Task 703 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 703 submitted for App characterizeImage, waiting on task 702
parsl.dataflow.dflow DEBUG: Task 703 set to pending state with AppFuture: <AppFuture at 0x14aa1582ff90 state=pending>
parsl.dataflow.dflow DEBUG: Task 703 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 704 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 704 submitted for App calibrate, waiting on task 703
parsl.dataflow.dflow DEBUG: Task 704 set to pending state with AppFuture: <AppFuture at 0x14aa15834390 state=pending>
parsl.dataflow.dflow DEBUG: Task 704 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 705 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 705 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 705 set to pending state with AppFuture: <AppFuture at 0x14aa15835010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 705 has memoization hash ee92226c1f216372a44a5eefde20580b
parsl.dataflow.memoization INFO: Task 705 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 705 try 0 launched on executor multi with executor id 236
parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/39beb8d6-9d7c-4a69-b9ea-1b9189f489e3_isr_855820_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/39beb8d6-9d7c-4a69-b9ea-1b9189f489e3_isr_855820_23.stderr
parsl.dataflow.dflow DEBUG: Task 706 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 706 submitted for App characterizeImage, waiting on task 705
parsl.dataflow.dflow DEBUG: Task 706 set to pending state with AppFuture: <AppFuture at 0x14aa15834910 state=pending>
parsl.dataflow.dflow DEBUG: Task 706 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 707 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 707 submitted for App calibrate, waiting on task 706
parsl.dataflow.dflow DEBUG: Task 707 set to pending state with AppFuture: <AppFuture at 0x14aa15835e90 state=pending>
parsl.dataflow.dflow DEBUG: Task 707 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 708 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 708 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 708 set to pending state with AppFuture: <AppFuture at 0x14aa15836490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 708 has memoization hash 2295a722db5698bcca473cc9dea3a5a6
parsl.dataflow.memoization INFO: Task 708 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 708 try 0 launched on executor multi with executor id 237
parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/39240542-7563-4c3e-8888-3f266876b357_isr_855910_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/39240542-7563-4c3e-8888-3f266876b357_isr_855910_40.stderr
parsl.dataflow.dflow DEBUG: Task 709 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 709 submitted for App characterizeImage, waiting on task 708
parsl.dataflow.dflow DEBUG: Task 709 set to pending state with AppFuture: <AppFuture at 0x14aa15836d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 709 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 710 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 710 submitted for App calibrate, waiting on task 709
parsl.dataflow.dflow DEBUG: Task 710 set to pending state with AppFuture: <AppFuture at 0x14aa15836c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 710 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 711 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 711 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 711 set to pending state with AppFuture: <AppFuture at 0x14aa158378d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 711 has memoization hash c246af6af7ef788aeda0e5a3f37b583b
parsl.dataflow.memoization INFO: Task 711 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 711 try 0 launched on executor multi with executor id 238
parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/127ea4be-46a8-4527-88d5-8fed1d0af41c_isr_855910_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/127ea4be-46a8-4527-88d5-8fed1d0af41c_isr_855910_13.stderr
parsl.dataflow.dflow DEBUG: Task 712 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 712 submitted for App characterizeImage, waiting on task 711
parsl.dataflow.dflow DEBUG: Task 712 set to pending state with AppFuture: <AppFuture at 0x14aa15837e10 state=pending>
parsl.dataflow.dflow DEBUG: Task 712 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 713 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 713 submitted for App calibrate, waiting on task 712
parsl.dataflow.dflow DEBUG: Task 713 set to pending state with AppFuture: <AppFuture at 0x14aa15840710 state=pending>
parsl.dataflow.dflow DEBUG: Task 713 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 714 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 714 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 714 set to pending state with AppFuture: <AppFuture at 0x14aa15840cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 714 has memoization hash 3818d870b24ce700ba9df21d23da87d3
parsl.dataflow.memoization INFO: Task 714 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 714 try 0 launched on executor multi with executor id 239
parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/36fd6a43-4c21-4633-b7e9-c357a4226bb8_isr_855821_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/36fd6a43-4c21-4633-b7e9-c357a4226bb8_isr_855821_14.stderr
parsl.dataflow.dflow DEBUG: Task 715 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 715 submitted for App characterizeImage, waiting on task 714
parsl.dataflow.dflow DEBUG: Task 715 set to pending state with AppFuture: <AppFuture at 0x14aa158405d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 715 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 716 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 716 submitted for App calibrate, waiting on task 715
parsl.dataflow.dflow DEBUG: Task 716 set to pending state with AppFuture: <AppFuture at 0x14aa158415d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 716 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 717 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 717 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 717 set to pending state with AppFuture: <AppFuture at 0x14aa15842090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 717 has memoization hash c020376f865a232d431c52d9f47dfc6e
parsl.dataflow.memoization INFO: Task 717 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 717 try 0 launched on executor multi with executor id 240
parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/52ab0021-3fa0-4fa4-8c7e-c3f72e2c182d_isr_855820_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/52ab0021-3fa0-4fa4-8c7e-c3f72e2c182d_isr_855820_41.stderr
parsl.dataflow.dflow DEBUG: Task 718 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 718 submitted for App characterizeImage, waiting on task 717
parsl.dataflow.dflow DEBUG: Task 718 set to pending state with AppFuture: <AppFuture at 0x14aa15841510 state=pending>
parsl.dataflow.dflow DEBUG: Task 718 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 719 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 719 submitted for App calibrate, waiting on task 718
parsl.dataflow.dflow DEBUG: Task 719 set to pending state with AppFuture: <AppFuture at 0x14aa15843010 state=pending>
parsl.dataflow.dflow DEBUG: Task 719 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 720 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 720 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 720 set to pending state with AppFuture: <AppFuture at 0x14aa15843610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 720 has memoization hash 795bae18b5ccbda5bddf19d4592a4427
parsl.dataflow.memoization INFO: Task 720 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 720 try 0 launched on executor multi with executor id 241
parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/0c350f46-9160-4d48-8514-feff4707ab35_isr_855911_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/0c350f46-9160-4d48-8514-feff4707ab35_isr_855911_13.stderr
parsl.dataflow.dflow DEBUG: Task 721 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 721 submitted for App characterizeImage, waiting on task 720
parsl.dataflow.dflow DEBUG: Task 721 set to pending state with AppFuture: <AppFuture at 0x14aa15843350 state=pending>
parsl.dataflow.dflow DEBUG: Task 721 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 722 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 722 submitted for App calibrate, waiting on task 721
parsl.dataflow.dflow DEBUG: Task 722 set to pending state with AppFuture: <AppFuture at 0x14aa15850450 state=pending>
parsl.dataflow.dflow DEBUG: Task 722 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 723 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 723 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 723 set to pending state with AppFuture: <AppFuture at 0x14aa15850a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 723 has memoization hash 86406b7c904ccfd303fd0df5d667c907
parsl.dataflow.memoization INFO: Task 723 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 723 try 0 launched on executor multi with executor id 242
parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/02f66a2c-c6cd-408b-9739-7c0b039ad12b_isr_855911_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/02f66a2c-c6cd-408b-9739-7c0b039ad12b_isr_855911_6.stderr
parsl.dataflow.dflow DEBUG: Task 724 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 724 submitted for App characterizeImage, waiting on task 723
parsl.dataflow.dflow DEBUG: Task 724 set to pending state with AppFuture: <AppFuture at 0x14aa15850bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 724 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 725 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 725 submitted for App calibrate, waiting on task 724
parsl.dataflow.dflow DEBUG: Task 725 set to pending state with AppFuture: <AppFuture at 0x14aa15851290 state=pending>
parsl.dataflow.dflow DEBUG: Task 725 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 726 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 726 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 726 set to pending state with AppFuture: <AppFuture at 0x14aa15851e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 726 has memoization hash 218c0a3d1f7331fe926da744b9cfe89a
parsl.dataflow.memoization INFO: Task 726 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158580e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 726 try 0 launched on executor multi with executor id 243
parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/66aa9739-d896-4bf1-b7e9-8333bfa79df1_isr_855822_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/66aa9739-d896-4bf1-b7e9-8333bfa79df1_isr_855822_31.stderr
parsl.dataflow.dflow DEBUG: Task 727 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 727 submitted for App characterizeImage, waiting on task 726
parsl.dataflow.dflow DEBUG: Task 727 set to pending state with AppFuture: <AppFuture at 0x14aa158526d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 727 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 728 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 728 submitted for App calibrate, waiting on task 727
parsl.dataflow.dflow DEBUG: Task 728 set to pending state with AppFuture: <AppFuture at 0x14aa15852f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 728 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 729 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 729 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 729 set to pending state with AppFuture: <AppFuture at 0x14aa15853250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 729 has memoization hash 4216fc2de651cdf40e925f0ddec90bfd
parsl.dataflow.memoization INFO: Task 729 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15858400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 729 try 0 launched on executor multi with executor id 244
parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/e2e04629-acba-4e6c-b5cf-3c29f43909fc_isr_855911_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/e2e04629-acba-4e6c-b5cf-3c29f43909fc_isr_855911_26.stderr
parsl.dataflow.dflow DEBUG: Task 730 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 730 submitted for App characterizeImage, waiting on task 729
parsl.dataflow.dflow DEBUG: Task 730 set to pending state with AppFuture: <AppFuture at 0x14aa15853390 state=pending>
parsl.dataflow.dflow DEBUG: Task 730 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 731 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 731 submitted for App calibrate, waiting on task 730
parsl.dataflow.dflow DEBUG: Task 731 set to pending state with AppFuture: <AppFuture at 0x14aa15853cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 731 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 732 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 732 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 732 set to pending state with AppFuture: <AppFuture at 0x14aa1585c690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 732 has memoization hash d7019221d6788f0e057116089ab83088
parsl.dataflow.memoization INFO: Task 732 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15858720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 732 try 0 launched on executor multi with executor id 245
parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0bb11aa8-81cc-452c-b040-ef15ace97434_isr_855820_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0bb11aa8-81cc-452c-b040-ef15ace97434_isr_855820_30.stderr
parsl.dataflow.dflow DEBUG: Task 733 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 733 submitted for App characterizeImage, waiting on task 732
parsl.dataflow.dflow DEBUG: Task 733 set to pending state with AppFuture: <AppFuture at 0x14aa1b5ecf90 state=pending>
parsl.dataflow.dflow DEBUG: Task 733 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 734 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 734 submitted for App calibrate, waiting on task 733
parsl.dataflow.dflow DEBUG: Task 734 set to pending state with AppFuture: <AppFuture at 0x14aa1585d090 state=pending>
parsl.dataflow.dflow DEBUG: Task 734 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 735 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 735 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 735 set to pending state with AppFuture: <AppFuture at 0x14aa1585dad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 735 has memoization hash 2a750e03cc6ea07ae9c80b266dae77c9
parsl.dataflow.memoization INFO: Task 735 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15858a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 735 try 0 launched on executor multi with executor id 246
parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/1242aa15-e5c4-455a-8993-00b06d196ac7_isr_855821_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/1242aa15-e5c4-455a-8993-00b06d196ac7_isr_855821_27.stderr
parsl.dataflow.dflow DEBUG: Task 736 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 736 submitted for App characterizeImage, waiting on task 735
parsl.dataflow.dflow DEBUG: Task 736 set to pending state with AppFuture: <AppFuture at 0x14aa159cd050 state=pending>
parsl.dataflow.dflow DEBUG: Task 736 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 737 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 737 submitted for App calibrate, waiting on task 736
parsl.dataflow.dflow DEBUG: Task 737 set to pending state with AppFuture: <AppFuture at 0x14aa1585e990 state=pending>
parsl.dataflow.dflow DEBUG: Task 737 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 738 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 738 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 738 set to pending state with AppFuture: <AppFuture at 0x14aa1585ef10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 738 has memoization hash 37a5699334d9a1d0eb1b65293e2a0910
parsl.dataflow.memoization INFO: Task 738 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15858d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 738 try 0 launched on executor multi with executor id 247
parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/45ec7539-3cc6-4c20-9887-31ce526aac89_isr_855909_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/45ec7539-3cc6-4c20-9887-31ce526aac89_isr_855909_61.stderr
parsl.dataflow.dflow DEBUG: Task 739 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 739 submitted for App characterizeImage, waiting on task 738
parsl.dataflow.dflow DEBUG: Task 739 set to pending state with AppFuture: <AppFuture at 0x14aa1585f210 state=pending>
parsl.dataflow.dflow DEBUG: Task 739 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 740 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 740 submitted for App calibrate, waiting on task 739
parsl.dataflow.dflow DEBUG: Task 740 set to pending state with AppFuture: <AppFuture at 0x14aa1585ff10 state=pending>
parsl.dataflow.dflow DEBUG: Task 740 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 741 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 741 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 741 set to pending state with AppFuture: <AppFuture at 0x14aa15864350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 741 has memoization hash 3a74096c9f7af438940c537c18f4f1ec
parsl.dataflow.memoization INFO: Task 741 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15859080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 741 try 0 launched on executor multi with executor id 248
parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/84c5640b-7aa6-4199-a2dd-1aa3b03c68b3_isr_855910_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/84c5640b-7aa6-4199-a2dd-1aa3b03c68b3_isr_855910_3.stderr
parsl.dataflow.dflow DEBUG: Task 742 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 742 submitted for App characterizeImage, waiting on task 741
parsl.dataflow.dflow DEBUG: Task 742 set to pending state with AppFuture: <AppFuture at 0x14aa15864090 state=pending>
parsl.dataflow.dflow DEBUG: Task 742 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 743 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 743 submitted for App calibrate, waiting on task 742
parsl.dataflow.dflow DEBUG: Task 743 set to pending state with AppFuture: <AppFuture at 0x14aa15864ad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 743 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 744 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 744 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 744 set to pending state with AppFuture: <AppFuture at 0x14aa15865790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 744 has memoization hash e9026b88a57bc925b25ea1ff78426e85
parsl.dataflow.memoization INFO: Task 744 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158593a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 744 try 0 launched on executor multi with executor id 249
parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/7536f64a-7e03-48a3-b1df-d3b43eba886a_isr_855909_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/7536f64a-7e03-48a3-b1df-d3b43eba886a_isr_855909_55.stderr
parsl.dataflow.dflow DEBUG: Task 745 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 745 submitted for App characterizeImage, waiting on task 744
parsl.dataflow.dflow DEBUG: Task 745 set to pending state with AppFuture: <AppFuture at 0x14aa158654d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 745 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 746 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 746 submitted for App calibrate, waiting on task 745
parsl.dataflow.dflow DEBUG: Task 746 set to pending state with AppFuture: <AppFuture at 0x14aa15866610 state=pending>
parsl.dataflow.dflow DEBUG: Task 746 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 747 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 747 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 747 set to pending state with AppFuture: <AppFuture at 0x14aa15866c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 747 has memoization hash af000421b25ab3cf5b6f5bc7f29debd0
parsl.dataflow.memoization INFO: Task 747 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158596c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 747 try 0 launched on executor multi with executor id 250
parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/69013455-02bc-4ff6-a1c7-cda91ad3bdd6_isr_855821_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/69013455-02bc-4ff6-a1c7-cda91ad3bdd6_isr_855821_22.stderr
parsl.dataflow.dflow DEBUG: Task 748 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 748 submitted for App characterizeImage, waiting on task 747
parsl.dataflow.dflow DEBUG: Task 748 set to pending state with AppFuture: <AppFuture at 0x14aa15866950 state=pending>
parsl.dataflow.dflow DEBUG: Task 748 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 749 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 749 submitted for App calibrate, waiting on task 748
parsl.dataflow.dflow DEBUG: Task 749 set to pending state with AppFuture: <AppFuture at 0x14aa15867c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 749 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 750 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 750 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 750 set to pending state with AppFuture: <AppFuture at 0x14aa15867e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 750 has memoization hash d777f60b0f27777b2c61161f81281d29
parsl.dataflow.memoization INFO: Task 750 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158599e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 750 try 0 launched on executor multi with executor id 251
parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/f2de0941-d11d-4dd2-a7f7-29a05f7cfed9_isr_855911_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/f2de0941-d11d-4dd2-a7f7-29a05f7cfed9_isr_855911_46.stderr
parsl.dataflow.dflow DEBUG: Task 751 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 751 submitted for App characterizeImage, waiting on task 750
parsl.dataflow.dflow DEBUG: Task 751 set to pending state with AppFuture: <AppFuture at 0x14aa15867c50 state=pending>
parsl.dataflow.dflow DEBUG: Task 751 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 752 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 752 submitted for App calibrate, waiting on task 751
parsl.dataflow.dflow DEBUG: Task 752 set to pending state with AppFuture: <AppFuture at 0x14aa15870f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 752 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 753 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 753 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 753 set to pending state with AppFuture: <AppFuture at 0x14aa15871490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 753 has memoization hash bcd31f770cf139c618e4eb05722843d2
parsl.dataflow.memoization INFO: Task 753 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15859d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 753 try 0 launched on executor multi with executor id 252
parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/b09f437c-d848-4f92-8aa5-1da8a13470ef_isr_855822_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/b09f437c-d848-4f92-8aa5-1da8a13470ef_isr_855822_47.stderr
parsl.dataflow.dflow DEBUG: Task 754 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 754 submitted for App characterizeImage, waiting on task 753
parsl.dataflow.dflow DEBUG: Task 754 set to pending state with AppFuture: <AppFuture at 0x14aa15870c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 754 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 755 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 755 submitted for App calibrate, waiting on task 754
parsl.dataflow.dflow DEBUG: Task 755 set to pending state with AppFuture: <AppFuture at 0x14aa15871ed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 755 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 756 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 756 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 756 set to pending state with AppFuture: <AppFuture at 0x14aa15872890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 756 has memoization hash c0fa94ba829e9c77388e6af973adbf49
parsl.dataflow.memoization INFO: Task 756 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 756 try 0 launched on executor multi with executor id 253
parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/b3606f6b-4001-4028-85c7-7c58e468c43f_isr_855910_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/b3606f6b-4001-4028-85c7-7c58e468c43f_isr_855910_12.stderr
parsl.dataflow.dflow DEBUG: Task 757 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 757 submitted for App characterizeImage, waiting on task 756
parsl.dataflow.dflow DEBUG: Task 757 set to pending state with AppFuture: <AppFuture at 0x14aa15873490 state=pending>
parsl.dataflow.dflow DEBUG: Task 757 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 758 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 758 submitted for App calibrate, waiting on task 757
parsl.dataflow.dflow DEBUG: Task 758 set to pending state with AppFuture: <AppFuture at 0x14aa15873650 state=pending>
parsl.dataflow.dflow DEBUG: Task 758 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 759 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 759 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 759 set to pending state with AppFuture: <AppFuture at 0x14aa15873c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 759 has memoization hash 0893d23a4d693cc559b7b989623dd33d
parsl.dataflow.memoization INFO: Task 759 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 759 try 0 launched on executor multi with executor id 254
parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/2672b821-e17b-4ecb-b838-5e4577fdcf4b_isr_855910_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/2672b821-e17b-4ecb-b838-5e4577fdcf4b_isr_855910_30.stderr
parsl.dataflow.dflow DEBUG: Task 760 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 760 submitted for App characterizeImage, waiting on task 759
parsl.dataflow.dflow DEBUG: Task 760 set to pending state with AppFuture: <AppFuture at 0x14aa157f4310 state=pending>
parsl.dataflow.dflow DEBUG: Task 760 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 761 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 761 submitted for App calibrate, waiting on task 760
parsl.dataflow.dflow DEBUG: Task 761 set to pending state with AppFuture: <AppFuture at 0x14aa1587cad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 761 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 762 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 762 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 762 set to pending state with AppFuture: <AppFuture at 0x14aa1587d090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 762 has memoization hash 3dd28856e9aa44e8a23b1e5ef70cca19
parsl.dataflow.memoization INFO: Task 762 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 762 try 0 launched on executor multi with executor id 255
parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/3619b24e-5d96-4913-9877-cf3d4700d511_isr_855821_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/3619b24e-5d96-4913-9877-cf3d4700d511_isr_855821_2.stderr
parsl.dataflow.dflow DEBUG: Task 763 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 763 submitted for App characterizeImage, waiting on task 762
parsl.dataflow.dflow DEBUG: Task 763 set to pending state with AppFuture: <AppFuture at 0x14aa1587c5d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 763 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 764 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 764 submitted for App calibrate, waiting on task 763
parsl.dataflow.dflow DEBUG: Task 764 set to pending state with AppFuture: <AppFuture at 0x14aa1587d890 state=pending>
parsl.dataflow.dflow DEBUG: Task 764 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 765 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 765 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 765 set to pending state with AppFuture: <AppFuture at 0x14aa1587e410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 765 has memoization hash cc8e22486c46a81bb2dc63de2ed7bb3e
parsl.dataflow.memoization INFO: Task 765 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 765 try 0 launched on executor multi with executor id 256
parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/d9ebb859-4b60-4478-8874-3d8719c239cd_isr_855910_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/d9ebb859-4b60-4478-8874-3d8719c239cd_isr_855910_59.stderr
parsl.dataflow.dflow DEBUG: Task 766 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 766 submitted for App characterizeImage, waiting on task 765
parsl.dataflow.dflow DEBUG: Task 766 set to pending state with AppFuture: <AppFuture at 0x14aa1587e150 state=pending>
parsl.dataflow.dflow DEBUG: Task 766 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 767 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 767 submitted for App calibrate, waiting on task 766
parsl.dataflow.dflow DEBUG: Task 767 set to pending state with AppFuture: <AppFuture at 0x14aa1587f490 state=pending>
parsl.dataflow.dflow DEBUG: Task 767 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 768 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 768 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 768 set to pending state with AppFuture: <AppFuture at 0x14aa1587f890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 768 has memoization hash 220e8a179aa28e7221683b7904412504
parsl.dataflow.memoization INFO: Task 768 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 768 try 0 launched on executor multi with executor id 257
parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/22173b18-8790-4209-8d6c-3b8dfd695d74_isr_855821_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/22173b18-8790-4209-8d6c-3b8dfd695d74_isr_855821_43.stderr
parsl.dataflow.dflow DEBUG: Task 769 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 769 submitted for App characterizeImage, waiting on task 768
parsl.dataflow.dflow DEBUG: Task 769 set to pending state with AppFuture: <AppFuture at 0x14aa1587f5d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 769 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 770 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 770 submitted for App calibrate, waiting on task 769
parsl.dataflow.dflow DEBUG: Task 770 set to pending state with AppFuture: <AppFuture at 0x14aa15880810 state=pending>
parsl.dataflow.dflow DEBUG: Task 770 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 771 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 771 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 771 set to pending state with AppFuture: <AppFuture at 0x14aa15880c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 771 has memoization hash 93616cefb1503a7a636805517a81237a
parsl.dataflow.memoization INFO: Task 771 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 771 try 0 launched on executor multi with executor id 258
parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/74bf6a4e-f186-47f4-be21-abb6012e4609_isr_855820_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/74bf6a4e-f186-47f4-be21-abb6012e4609_isr_855820_31.stderr
parsl.dataflow.dflow DEBUG: Task 772 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 772 submitted for App characterizeImage, waiting on task 771
parsl.dataflow.dflow DEBUG: Task 772 set to pending state with AppFuture: <AppFuture at 0x14aa15880050 state=pending>
parsl.dataflow.dflow DEBUG: Task 772 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 773 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 773 submitted for App calibrate, waiting on task 772
parsl.dataflow.dflow DEBUG: Task 773 set to pending state with AppFuture: <AppFuture at 0x14aa15881650 state=pending>
parsl.dataflow.dflow DEBUG: Task 773 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 774 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 774 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 774 set to pending state with AppFuture: <AppFuture at 0x14aa15882050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 774 has memoization hash 99f22b5276dacaefeaffbf55dd3987dd
parsl.dataflow.memoization INFO: Task 774 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 774 try 0 launched on executor multi with executor id 259
parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/88dab2ac-c3c1-40b0-8adf-1f63c055c027_isr_855822_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/88dab2ac-c3c1-40b0-8adf-1f63c055c027_isr_855822_26.stderr
parsl.dataflow.dflow DEBUG: Task 775 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 775 submitted for App characterizeImage, waiting on task 774
parsl.dataflow.dflow DEBUG: Task 775 set to pending state with AppFuture: <AppFuture at 0x14aa15881d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 775 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 776 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 776 submitted for App calibrate, waiting on task 775
parsl.dataflow.dflow DEBUG: Task 776 set to pending state with AppFuture: <AppFuture at 0x14aa15883050 state=pending>
parsl.dataflow.dflow DEBUG: Task 776 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 777 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 777 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 777 set to pending state with AppFuture: <AppFuture at 0x14aa15883550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 777 has memoization hash 8f6b0838b99f8bf1b52897f551e7f125
parsl.dataflow.memoization INFO: Task 777 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 777 try 0 launched on executor multi with executor id 260
parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/d765b95b-58af-4a43-bfea-95d99333737c_isr_855909_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/d765b95b-58af-4a43-bfea-95d99333737c_isr_855909_8.stderr
parsl.dataflow.dflow DEBUG: Task 778 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 778 submitted for App characterizeImage, waiting on task 777
parsl.dataflow.dflow DEBUG: Task 778 set to pending state with AppFuture: <AppFuture at 0x14aa15883690 state=pending>
parsl.dataflow.dflow DEBUG: Task 778 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 779 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 779 submitted for App calibrate, waiting on task 778
parsl.dataflow.dflow DEBUG: Task 779 set to pending state with AppFuture: <AppFuture at 0x14aa158903d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 779 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 780 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 780 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 780 set to pending state with AppFuture: <AppFuture at 0x14aa158909d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 780 has memoization hash 8938f3ca7c56d6a4e6b1211c5fe98194
parsl.dataflow.memoization INFO: Task 780 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 780 try 0 launched on executor multi with executor id 261
parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/f264fae1-3a01-43a0-b6b7-20c31e05c08e_isr_855911_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/f264fae1-3a01-43a0-b6b7-20c31e05c08e_isr_855911_24.stderr
parsl.dataflow.dflow DEBUG: Task 781 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 781 submitted for App characterizeImage, waiting on task 780
parsl.dataflow.dflow DEBUG: Task 781 set to pending state with AppFuture: <AppFuture at 0x14aa1b5f9010 state=pending>
parsl.dataflow.dflow DEBUG: Task 781 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 782 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 782 submitted for App calibrate, waiting on task 781
parsl.dataflow.dflow DEBUG: Task 782 set to pending state with AppFuture: <AppFuture at 0x14aa158913d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 782 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 783 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 783 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 783 set to pending state with AppFuture: <AppFuture at 0x14aa15891d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 783 has memoization hash 63449c431d7ef07a6786c7fd98250efc
parsl.dataflow.memoization INFO: Task 783 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 783 try 0 launched on executor multi with executor id 262
parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/c9689da1-0737-45f0-a1eb-508acbae4e1d_isr_855821_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/c9689da1-0737-45f0-a1eb-508acbae4e1d_isr_855821_23.stderr
parsl.dataflow.dflow DEBUG: Task 784 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 784 submitted for App characterizeImage, waiting on task 783
parsl.dataflow.dflow DEBUG: Task 784 set to pending state with AppFuture: <AppFuture at 0x14aa15891a90 state=pending>
parsl.dataflow.dflow DEBUG: Task 784 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 785 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 785 submitted for App calibrate, waiting on task 784
parsl.dataflow.dflow DEBUG: Task 785 set to pending state with AppFuture: <AppFuture at 0x14aa15892c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 785 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 786 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 786 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 786 set to pending state with AppFuture: <AppFuture at 0x14aa15893150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 786 has memoization hash a69df3ba58fbd261b9ddee2ddaa521b2
parsl.dataflow.memoization INFO: Task 786 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 786 try 0 launched on executor multi with executor id 263
parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/257922e6-6174-4f1a-8326-33f93668e4fd_isr_855821_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/257922e6-6174-4f1a-8326-33f93668e4fd_isr_855821_29.stderr
parsl.dataflow.dflow DEBUG: Task 787 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 787 submitted for App characterizeImage, waiting on task 786
parsl.dataflow.dflow DEBUG: Task 787 set to pending state with AppFuture: <AppFuture at 0x14aa15892390 state=pending>
parsl.dataflow.dflow DEBUG: Task 787 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 788 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 788 submitted for App calibrate, waiting on task 787
parsl.dataflow.dflow DEBUG: Task 788 set to pending state with AppFuture: <AppFuture at 0x14aa15893550 state=pending>
parsl.dataflow.dflow DEBUG: Task 788 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 789 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 789 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 789 set to pending state with AppFuture: <AppFuture at 0x14aa1589c550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 789 has memoization hash f2ef1f2feb4d61ffbc590d35205b0ca9
parsl.dataflow.memoization INFO: Task 789 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158942c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 789 try 0 launched on executor multi with executor id 264
parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/aa55c955-2470-49d5-bc6e-1cfaab428703_isr_855910_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/aa55c955-2470-49d5-bc6e-1cfaab428703_isr_855910_53.stderr
parsl.dataflow.dflow DEBUG: Task 790 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 790 submitted for App characterizeImage, waiting on task 789
parsl.dataflow.dflow DEBUG: Task 790 set to pending state with AppFuture: <AppFuture at 0x14aa1589c690 state=pending>
parsl.dataflow.dflow DEBUG: Task 790 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 791 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 791 submitted for App calibrate, waiting on task 790
parsl.dataflow.dflow DEBUG: Task 791 set to pending state with AppFuture: <AppFuture at 0x14aa1589d390 state=pending>
parsl.dataflow.dflow DEBUG: Task 791 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 792 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 792 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 792 set to pending state with AppFuture: <AppFuture at 0x14aa1589d990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 792 has memoization hash 2e954ad5dcb2e382f78aa3b532423eba
parsl.dataflow.memoization INFO: Task 792 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158945e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 792 try 0 launched on executor multi with executor id 265
parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/7a549c9a-8980-4e2e-9b0b-afa1ab105449_isr_855911_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/7a549c9a-8980-4e2e-9b0b-afa1ab105449_isr_855911_40.stderr
parsl.dataflow.dflow DEBUG: Task 793 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 793 submitted for App characterizeImage, waiting on task 792
parsl.dataflow.dflow DEBUG: Task 793 set to pending state with AppFuture: <AppFuture at 0x14aa1589d6d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 793 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 794 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 794 submitted for App calibrate, waiting on task 793
parsl.dataflow.dflow DEBUG: Task 794 set to pending state with AppFuture: <AppFuture at 0x14aa1589e7d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 794 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 795 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 795 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 795 set to pending state with AppFuture: <AppFuture at 0x14aa1589edd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 795 has memoization hash 2e004977c5663572995a3a1fde9ea3c2
parsl.dataflow.memoization INFO: Task 795 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15894900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 795 try 0 launched on executor multi with executor id 266
parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/38d16b36-3eb8-4cb1-93a5-b8b065e85842_isr_855820_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/38d16b36-3eb8-4cb1-93a5-b8b065e85842_isr_855820_50.stderr
parsl.dataflow.dflow DEBUG: Task 796 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 796 submitted for App characterizeImage, waiting on task 795
parsl.dataflow.dflow DEBUG: Task 796 set to pending state with AppFuture: <AppFuture at 0x14aa1589cf50 state=pending>
parsl.dataflow.dflow DEBUG: Task 796 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 797 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 797 submitted for App calibrate, waiting on task 796
parsl.dataflow.dflow DEBUG: Task 797 set to pending state with AppFuture: <AppFuture at 0x14aa1589fe90 state=pending>
parsl.dataflow.dflow DEBUG: Task 797 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 798 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 798 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 798 set to pending state with AppFuture: <AppFuture at 0x14aa158a8110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 798 has memoization hash 2cc151ebe8d7d14ef1369a5471097e92
parsl.dataflow.memoization INFO: Task 798 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15894c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 798 try 0 launched on executor multi with executor id 267
parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/870fedf0-b3ff-48bb-a501-da12b5771fe1_isr_855909_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/870fedf0-b3ff-48bb-a501-da12b5771fe1_isr_855909_3.stderr
parsl.dataflow.dflow DEBUG: Task 799 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 799 submitted for App characterizeImage, waiting on task 798
parsl.dataflow.dflow DEBUG: Task 799 set to pending state with AppFuture: <AppFuture at 0x14aa1589fe10 state=pending>
parsl.dataflow.dflow DEBUG: Task 799 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 800 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 800 submitted for App calibrate, waiting on task 799
parsl.dataflow.dflow DEBUG: Task 800 set to pending state with AppFuture: <AppFuture at 0x14aa158a9110 state=pending>
parsl.dataflow.dflow DEBUG: Task 800 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 801 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 801 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 801 set to pending state with AppFuture: <AppFuture at 0x14aa158a9610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 801 has memoization hash e2cc255f7cb3512da79041b1f4ec486e
parsl.dataflow.memoization INFO: Task 801 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15894f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 801 try 0 launched on executor multi with executor id 268
parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/93ed9b76-d6e9-4892-a484-65151bab292e_isr_855910_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/93ed9b76-d6e9-4892-a484-65151bab292e_isr_855910_39.stderr
parsl.dataflow.dflow DEBUG: Task 802 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 802 submitted for App characterizeImage, waiting on task 801
parsl.dataflow.dflow DEBUG: Task 802 set to pending state with AppFuture: <AppFuture at 0x14aa158a9350 state=pending>
parsl.dataflow.dflow DEBUG: Task 802 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 803 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 803 submitted for App calibrate, waiting on task 802
parsl.dataflow.dflow DEBUG: Task 803 set to pending state with AppFuture: <AppFuture at 0x14aa158aa490 state=pending>
parsl.dataflow.dflow DEBUG: Task 803 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 804 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 804 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 804 set to pending state with AppFuture: <AppFuture at 0x14aa158aab50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 804 has memoization hash 9cdf70246072a3645efd02b6161307d3
parsl.dataflow.memoization INFO: Task 804 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15895260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 804 try 0 launched on executor multi with executor id 269
parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/ace894e0-3704-445b-97c5-b2173f2fcae1_isr_855821_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/ace894e0-3704-445b-97c5-b2173f2fcae1_isr_855821_26.stderr
parsl.dataflow.dflow DEBUG: Task 805 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 805 submitted for App characterizeImage, waiting on task 804
parsl.dataflow.dflow DEBUG: Task 805 set to pending state with AppFuture: <AppFuture at 0x14aa158a9f50 state=pending>
parsl.dataflow.dflow DEBUG: Task 805 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 806 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 806 submitted for App calibrate, waiting on task 805
parsl.dataflow.dflow DEBUG: Task 806 set to pending state with AppFuture: <AppFuture at 0x14aa158aba50 state=pending>
parsl.dataflow.dflow DEBUG: Task 806 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 807 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 807 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 807 set to pending state with AppFuture: <AppFuture at 0x14aa158abf10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 807 has memoization hash 25c18acce806e87dc2b824647ffbad44
parsl.dataflow.memoization INFO: Task 807 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15895580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 807 try 0 launched on executor multi with executor id 270
parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/e8911895-0ef8-4fb8-9282-06daed1ffff6_isr_855910_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/e8911895-0ef8-4fb8-9282-06daed1ffff6_isr_855910_44.stderr
parsl.dataflow.dflow DEBUG: Task 808 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 808 submitted for App characterizeImage, waiting on task 807
parsl.dataflow.dflow DEBUG: Task 808 set to pending state with AppFuture: <AppFuture at 0x14aa158abbd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 808 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 809 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 809 submitted for App calibrate, waiting on task 808
parsl.dataflow.dflow DEBUG: Task 809 set to pending state with AppFuture: <AppFuture at 0x14aa158b4610 state=pending>
parsl.dataflow.dflow DEBUG: Task 809 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 810 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 810 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 810 set to pending state with AppFuture: <AppFuture at 0x14aa158b5350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 810 has memoization hash 34c34a15909074867e581dedc1d26c23
parsl.dataflow.memoization INFO: Task 810 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158958a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 810 try 0 launched on executor multi with executor id 271
parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/ba3b59c5-4ca5-4afd-930c-89be90c56dda_isr_855822_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/ba3b59c5-4ca5-4afd-930c-89be90c56dda_isr_855822_12.stderr
parsl.dataflow.dflow DEBUG: Task 811 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 811 submitted for App characterizeImage, waiting on task 810
parsl.dataflow.dflow DEBUG: Task 811 set to pending state with AppFuture: <AppFuture at 0x14aa158b46d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 811 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 812 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 812 submitted for App calibrate, waiting on task 811
parsl.dataflow.dflow DEBUG: Task 812 set to pending state with AppFuture: <AppFuture at 0x14aa158b61d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 812 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 813 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 813 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 813 set to pending state with AppFuture: <AppFuture at 0x14aa158b6710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 813 has memoization hash 00dd9ff12ff07d73585f5701b6357058
parsl.dataflow.memoization INFO: Task 813 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15895bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 813 try 0 launched on executor multi with executor id 272
parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/bdfa5a9c-b1e0-41d2-b3f7-6c4f5912873c_isr_855821_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/bdfa5a9c-b1e0-41d2-b3f7-6c4f5912873c_isr_855821_25.stderr
parsl.dataflow.dflow DEBUG: Task 814 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 814 submitted for App characterizeImage, waiting on task 813
parsl.dataflow.dflow DEBUG: Task 814 set to pending state with AppFuture: <AppFuture at 0x14aa158b6a10 state=pending>
parsl.dataflow.dflow DEBUG: Task 814 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 815 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 815 submitted for App calibrate, waiting on task 814
parsl.dataflow.dflow DEBUG: Task 815 set to pending state with AppFuture: <AppFuture at 0x14aa158b7610 state=pending>
parsl.dataflow.dflow DEBUG: Task 815 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 816 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 816 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 816 set to pending state with AppFuture: <AppFuture at 0x14aa158b7c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 816 has memoization hash da2318befdf94b972210c76213407f7a
parsl.dataflow.memoization INFO: Task 816 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15895ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 816 try 0 launched on executor multi with executor id 273
parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0d3db246-cee8-4a78-8b89-6bf1dd319a3c_isr_855820_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0d3db246-cee8-4a78-8b89-6bf1dd319a3c_isr_855820_40.stderr
parsl.dataflow.dflow DEBUG: Task 817 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 817 submitted for App characterizeImage, waiting on task 816
parsl.dataflow.dflow DEBUG: Task 817 set to pending state with AppFuture: <AppFuture at 0x14aa158b7f50 state=pending>
parsl.dataflow.dflow DEBUG: Task 817 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 818 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 818 submitted for App calibrate, waiting on task 817
parsl.dataflow.dflow DEBUG: Task 818 set to pending state with AppFuture: <AppFuture at 0x14aa158bc410 state=pending>
parsl.dataflow.dflow DEBUG: Task 818 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 819 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 819 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 819 set to pending state with AppFuture: <AppFuture at 0x14aa158bcf90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 819 has memoization hash f657b853ffcd0730dbdbb501d803983f
parsl.dataflow.memoization INFO: Task 819 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15896200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 819 try 0 launched on executor multi with executor id 274
parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/5863efc6-5e06-4718-b16b-48617da16d2d_isr_855911_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/5863efc6-5e06-4718-b16b-48617da16d2d_isr_855911_50.stderr
parsl.dataflow.dflow DEBUG: Task 820 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 820 submitted for App characterizeImage, waiting on task 819
parsl.dataflow.dflow DEBUG: Task 820 set to pending state with AppFuture: <AppFuture at 0x14aa158bd610 state=pending>
parsl.dataflow.dflow DEBUG: Task 820 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 821 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 821 submitted for App calibrate, waiting on task 820
parsl.dataflow.dflow DEBUG: Task 821 set to pending state with AppFuture: <AppFuture at 0x14aa158bd990 state=pending>
parsl.dataflow.dflow DEBUG: Task 821 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 822 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 822 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 822 set to pending state with AppFuture: <AppFuture at 0x14aa158be390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 822 has memoization hash 023f23a0cdee12fb623e971a57839ec3
parsl.dataflow.memoization INFO: Task 822 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15896520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 822 try 0 launched on executor multi with executor id 275
parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/72d15fde-1e6b-4f59-9931-eb1620cfeb22_isr_855911_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/72d15fde-1e6b-4f59-9931-eb1620cfeb22_isr_855911_20.stderr
parsl.dataflow.dflow DEBUG: Task 823 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 823 submitted for App characterizeImage, waiting on task 822
parsl.dataflow.dflow DEBUG: Task 823 set to pending state with AppFuture: <AppFuture at 0x14aa158bc390 state=pending>
parsl.dataflow.dflow DEBUG: Task 823 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 824 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 824 submitted for App calibrate, waiting on task 823
parsl.dataflow.dflow DEBUG: Task 824 set to pending state with AppFuture: <AppFuture at 0x14aa158bea10 state=pending>
parsl.dataflow.dflow DEBUG: Task 824 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 825 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 825 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 825 set to pending state with AppFuture: <AppFuture at 0x14aa158bf750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 825 has memoization hash 41e6a95a932b62838656526216fe34d8
parsl.dataflow.memoization INFO: Task 825 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15896840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 825 try 0 launched on executor multi with executor id 276
parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/6b8cfc49-c688-474d-9c9b-db5b00b80b85_isr_855820_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/6b8cfc49-c688-474d-9c9b-db5b00b80b85_isr_855820_56.stderr
parsl.dataflow.dflow DEBUG: Task 826 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 826 submitted for App characterizeImage, waiting on task 825
parsl.dataflow.dflow DEBUG: Task 826 set to pending state with AppFuture: <AppFuture at 0x14aa158bfd90 state=pending>
parsl.dataflow.dflow DEBUG: Task 826 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 827 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 827 submitted for App calibrate, waiting on task 826
parsl.dataflow.dflow DEBUG: Task 827 set to pending state with AppFuture: <AppFuture at 0x14aa158c45d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 827 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 828 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 828 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 828 set to pending state with AppFuture: <AppFuture at 0x14aa158c4b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 828 has memoization hash 129e71ae831c765629d2c35671ee35e7
parsl.dataflow.memoization INFO: Task 828 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15896b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 828 try 0 launched on executor multi with executor id 277
parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/8233cefd-094d-4686-becb-c052446e79b7_isr_855821_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/8233cefd-094d-4686-becb-c052446e79b7_isr_855821_47.stderr
parsl.dataflow.dflow DEBUG: Task 829 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 829 submitted for App characterizeImage, waiting on task 828
parsl.dataflow.dflow DEBUG: Task 829 set to pending state with AppFuture: <AppFuture at 0x14aa158c4210 state=pending>
parsl.dataflow.dflow DEBUG: Task 829 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 830 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 830 submitted for App calibrate, waiting on task 829
parsl.dataflow.dflow DEBUG: Task 830 set to pending state with AppFuture: <AppFuture at 0x14aa158c5650 state=pending>
parsl.dataflow.dflow DEBUG: Task 830 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 831 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 831 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 831 set to pending state with AppFuture: <AppFuture at 0x14aa158c6010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 831 has memoization hash 10d011b734a1086a838d19a2a08c3320
parsl.dataflow.memoization INFO: Task 831 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15896e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 831 try 0 launched on executor multi with executor id 278
parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/8960334f-5330-49ea-95d3-0c6a61b656a6_isr_855910_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/8960334f-5330-49ea-95d3-0c6a61b656a6_isr_855910_14.stderr
parsl.dataflow.dflow DEBUG: Task 832 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 832 submitted for App characterizeImage, waiting on task 831
parsl.dataflow.dflow DEBUG: Task 832 set to pending state with AppFuture: <AppFuture at 0x14aa158c55d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 832 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 833 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 833 submitted for App calibrate, waiting on task 832
parsl.dataflow.dflow DEBUG: Task 833 set to pending state with AppFuture: <AppFuture at 0x14aa158c6e50 state=pending>
parsl.dataflow.dflow DEBUG: Task 833 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 834 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 834 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 834 set to pending state with AppFuture: <AppFuture at 0x14aa158c73d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 834 has memoization hash a205f83b1d4648bdba5fc6948aab8896
parsl.dataflow.memoization INFO: Task 834 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158971a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 834 try 0 launched on executor multi with executor id 279
parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/21724746-21af-49bf-b740-1ddea7894017_isr_855909_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/21724746-21af-49bf-b740-1ddea7894017_isr_855909_57.stderr
parsl.dataflow.dflow DEBUG: Task 835 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 835 submitted for App characterizeImage, waiting on task 834
parsl.dataflow.dflow DEBUG: Task 835 set to pending state with AppFuture: <AppFuture at 0x14aa158c77d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 835 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 836 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 836 submitted for App calibrate, waiting on task 835
parsl.dataflow.dflow DEBUG: Task 836 set to pending state with AppFuture: <AppFuture at 0x14aa158c78d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 836 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 837 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 837 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 837 set to pending state with AppFuture: <AppFuture at 0x14aa158d4850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 837 has memoization hash fb6b7905d952f6caf75303793a3e5b5e
parsl.dataflow.memoization INFO: Task 837 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158974c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 837 try 0 launched on executor multi with executor id 280
parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/4f8aae4b-766b-4746-ae16-4971ff0113d8_isr_855909_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/4f8aae4b-766b-4746-ae16-4971ff0113d8_isr_855909_44.stderr
parsl.dataflow.dflow DEBUG: Task 838 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 838 submitted for App characterizeImage, waiting on task 837
parsl.dataflow.dflow DEBUG: Task 838 set to pending state with AppFuture: <AppFuture at 0x14aa158d5310 state=pending>
parsl.dataflow.dflow DEBUG: Task 838 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 839 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 839 submitted for App calibrate, waiting on task 838
parsl.dataflow.dflow DEBUG: Task 839 set to pending state with AppFuture: <AppFuture at 0x14aa158d5090 state=pending>
parsl.dataflow.dflow DEBUG: Task 839 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 840 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 840 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 840 set to pending state with AppFuture: <AppFuture at 0x14aa158d5c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 840 has memoization hash a551d02c2db46c4975d4bf21726593dd
parsl.dataflow.memoization INFO: Task 840 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158977e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 840 try 0 launched on executor multi with executor id 281
parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/9b1e7b79-7e11-41f1-9e64-6cf59602de77_isr_855821_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/9b1e7b79-7e11-41f1-9e64-6cf59602de77_isr_855821_42.stderr
parsl.dataflow.dflow DEBUG: Task 841 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 841 submitted for App characterizeImage, waiting on task 840
parsl.dataflow.dflow DEBUG: Task 841 set to pending state with AppFuture: <AppFuture at 0x14aa15836cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 841 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 842 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 842 submitted for App calibrate, waiting on task 841
parsl.dataflow.dflow DEBUG: Task 842 set to pending state with AppFuture: <AppFuture at 0x14aa158d68d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 842 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 843 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 843 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 843 set to pending state with AppFuture: <AppFuture at 0x14aa158d7050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 843 has memoization hash 03ef3eef05b4f025b520215242194265
parsl.dataflow.memoization INFO: Task 843 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15897b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 843 try 0 launched on executor multi with executor id 282
parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/6a43c1e1-c745-4ad0-90d5-3e920098cf91_isr_855909_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/6a43c1e1-c745-4ad0-90d5-3e920098cf91_isr_855909_23.stderr
parsl.dataflow.dflow DEBUG: Task 844 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 844 submitted for App characterizeImage, waiting on task 843
parsl.dataflow.dflow DEBUG: Task 844 set to pending state with AppFuture: <AppFuture at 0x14aa158d73d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 844 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 845 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 845 submitted for App calibrate, waiting on task 844
parsl.dataflow.dflow DEBUG: Task 845 set to pending state with AppFuture: <AppFuture at 0x14aa15818050 state=pending>
parsl.dataflow.dflow DEBUG: Task 845 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 846 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 846 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 846 set to pending state with AppFuture: <AppFuture at 0x14aa156e0490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 846 has memoization hash 7c307c55b37a95d290249555c06684b7
parsl.dataflow.memoization INFO: Task 846 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15897e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 846 try 0 launched on executor multi with executor id 283
parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/298e680c-2fe8-4dd6-8210-9ac53ec18774_isr_855910_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/298e680c-2fe8-4dd6-8210-9ac53ec18774_isr_855910_5.stderr
parsl.dataflow.dflow DEBUG: Task 847 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 847 submitted for App characterizeImage, waiting on task 846
parsl.dataflow.dflow DEBUG: Task 847 set to pending state with AppFuture: <AppFuture at 0x14aa158a8b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 847 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 848 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 848 submitted for App calibrate, waiting on task 847
parsl.dataflow.dflow DEBUG: Task 848 set to pending state with AppFuture: <AppFuture at 0x14aa156e0d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 848 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 849 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 849 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 849 set to pending state with AppFuture: <AppFuture at 0x14aa156e1910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 849 has memoization hash 227ccf6f7f88dc3a80845d81a7c512d9
parsl.dataflow.memoization INFO: Task 849 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 849 try 0 launched on executor multi with executor id 284
parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/31f31ed7-8c9d-463d-afda-a8ececb14448_isr_855909_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/31f31ed7-8c9d-463d-afda-a8ececb14448_isr_855909_13.stderr
parsl.dataflow.dflow DEBUG: Task 850 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 850 submitted for App characterizeImage, waiting on task 849
parsl.dataflow.dflow DEBUG: Task 850 set to pending state with AppFuture: <AppFuture at 0x14aa156e2090 state=pending>
parsl.dataflow.dflow DEBUG: Task 850 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 851 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 851 submitted for App calibrate, waiting on task 850
parsl.dataflow.dflow DEBUG: Task 851 set to pending state with AppFuture: <AppFuture at 0x14aa156e1e10 state=pending>
parsl.dataflow.dflow DEBUG: Task 851 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 852 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 852 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 852 set to pending state with AppFuture: <AppFuture at 0x14aa156e2dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 852 has memoization hash 161aba1f72a9e296ad322ae644c275db
parsl.dataflow.memoization INFO: Task 852 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 852 try 0 launched on executor multi with executor id 285
parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/1361dd39-4a1f-4a6a-88dc-2c0a209f0aeb_isr_855909_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/1361dd39-4a1f-4a6a-88dc-2c0a209f0aeb_isr_855909_62.stderr
parsl.dataflow.dflow DEBUG: Task 853 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 853 submitted for App characterizeImage, waiting on task 852
parsl.dataflow.dflow DEBUG: Task 853 set to pending state with AppFuture: <AppFuture at 0x14aa156e2110 state=pending>
parsl.dataflow.dflow DEBUG: Task 853 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 854 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 854 submitted for App calibrate, waiting on task 853
parsl.dataflow.dflow DEBUG: Task 854 set to pending state with AppFuture: <AppFuture at 0x14aa156e35d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 854 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 855 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 855 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 855 set to pending state with AppFuture: <AppFuture at 0x14aa156ec050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 855 has memoization hash 53fd3ff028234be2889ca1895eceddfb
parsl.dataflow.memoization INFO: Task 855 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 855 try 0 launched on executor multi with executor id 286
parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0e781439-cc63-4721-a37c-d401c2f90534_isr_855820_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0e781439-cc63-4721-a37c-d401c2f90534_isr_855820_35.stderr
parsl.dataflow.dflow DEBUG: Task 856 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 856 submitted for App characterizeImage, waiting on task 855
parsl.dataflow.dflow DEBUG: Task 856 set to pending state with AppFuture: <AppFuture at 0x14aa156e3c10 state=pending>
parsl.dataflow.dflow DEBUG: Task 856 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 857 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 857 submitted for App calibrate, waiting on task 856
parsl.dataflow.dflow DEBUG: Task 857 set to pending state with AppFuture: <AppFuture at 0x14aa156ec6d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 857 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 858 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 858 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 858 set to pending state with AppFuture: <AppFuture at 0x14aa156ed510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 858 has memoization hash 2f7027430a6c36dc4bc4ea1cd2e22f66
parsl.dataflow.memoization INFO: Task 858 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 858 try 0 launched on executor multi with executor id 287
parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/47e7974e-c8a0-418a-9029-0278f25ed9de_isr_855909_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/47e7974e-c8a0-418a-9029-0278f25ed9de_isr_855909_35.stderr
parsl.dataflow.dflow DEBUG: Task 859 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 859 submitted for App characterizeImage, waiting on task 858
parsl.dataflow.dflow DEBUG: Task 859 set to pending state with AppFuture: <AppFuture at 0x14aa156ecc50 state=pending>
parsl.dataflow.dflow DEBUG: Task 859 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 860 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 860 submitted for App calibrate, waiting on task 859
parsl.dataflow.dflow DEBUG: Task 860 set to pending state with AppFuture: <AppFuture at 0x14aa156ee390 state=pending>
parsl.dataflow.dflow DEBUG: Task 860 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 861 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 861 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 861 set to pending state with AppFuture: <AppFuture at 0x14aa156ee990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 861 has memoization hash f008c05cc2a84430d576180e167bb3b5
parsl.dataflow.memoization INFO: Task 861 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 861 try 0 launched on executor multi with executor id 288
parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/d6c6aa68-0427-4194-8fbe-16ab7173c96f_isr_855821_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/d6c6aa68-0427-4194-8fbe-16ab7173c96f_isr_855821_53.stderr
parsl.dataflow.dflow DEBUG: Task 862 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 862 submitted for App characterizeImage, waiting on task 861
parsl.dataflow.dflow DEBUG: Task 862 set to pending state with AppFuture: <AppFuture at 0x14aa156ee3d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 862 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 863 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 863 submitted for App calibrate, waiting on task 862
parsl.dataflow.dflow DEBUG: Task 863 set to pending state with AppFuture: <AppFuture at 0x14aa156ef210 state=pending>
parsl.dataflow.dflow DEBUG: Task 863 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 864 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 864 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 864 set to pending state with AppFuture: <AppFuture at 0x14aa156efd90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 864 has memoization hash 452ba0649c1a92593f114eb4d879b693
parsl.dataflow.memoization INFO: Task 864 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 864 try 0 launched on executor multi with executor id 289
parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/e511decf-a3d7-4477-9622-94d91e22ba5c_isr_855910_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/e511decf-a3d7-4477-9622-94d91e22ba5c_isr_855910_35.stderr
parsl.dataflow.dflow DEBUG: Task 865 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 865 submitted for App characterizeImage, waiting on task 864
parsl.dataflow.dflow DEBUG: Task 865 set to pending state with AppFuture: <AppFuture at 0x14aa156ef150 state=pending>
parsl.dataflow.dflow DEBUG: Task 865 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 866 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 866 submitted for App calibrate, waiting on task 865
parsl.dataflow.dflow DEBUG: Task 866 set to pending state with AppFuture: <AppFuture at 0x14aa156f88d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 866 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 867 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 867 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 867 set to pending state with AppFuture: <AppFuture at 0x14aa156f9210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 867 has memoization hash 23c34c11b27ca532ad83d348afe1187a
parsl.dataflow.memoization INFO: Task 867 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 867 try 0 launched on executor multi with executor id 290
parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/1f915565-a8a1-42ae-8fb5-262cdfbdc8d7_isr_855822_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/1f915565-a8a1-42ae-8fb5-262cdfbdc8d7_isr_855822_58.stderr
parsl.dataflow.dflow DEBUG: Task 868 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 868 submitted for App characterizeImage, waiting on task 867
parsl.dataflow.dflow DEBUG: Task 868 set to pending state with AppFuture: <AppFuture at 0x14aa156f8b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 868 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 869 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 869 submitted for App calibrate, waiting on task 868
parsl.dataflow.dflow DEBUG: Task 869 set to pending state with AppFuture: <AppFuture at 0x14aa156fa050 state=pending>
parsl.dataflow.dflow DEBUG: Task 869 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 870 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 870 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 870 set to pending state with AppFuture: <AppFuture at 0x14aa156fa590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 870 has memoization hash 600b00667825d193caab48278a3132eb
parsl.dataflow.memoization INFO: Task 870 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 870 try 0 launched on executor multi with executor id 291
parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/9813a636-51b6-40c5-b797-d49eda87b010_isr_855822_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/9813a636-51b6-40c5-b797-d49eda87b010_isr_855822_32.stderr
parsl.dataflow.dflow DEBUG: Task 871 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 871 submitted for App characterizeImage, waiting on task 870
parsl.dataflow.dflow DEBUG: Task 871 set to pending state with AppFuture: <AppFuture at 0x14aa156fa210 state=pending>
parsl.dataflow.dflow DEBUG: Task 871 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 872 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 872 submitted for App calibrate, waiting on task 871
parsl.dataflow.dflow DEBUG: Task 872 set to pending state with AppFuture: <AppFuture at 0x14aa156fb5d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 872 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 873 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 873 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 873 set to pending state with AppFuture: <AppFuture at 0x14aa156fb9d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 873 has memoization hash 090185391210d609d94164a08c269b45
parsl.dataflow.memoization INFO: Task 873 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 873 try 0 launched on executor multi with executor id 292
parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/8623eb92-f4fd-4c1f-a6cd-b27ee2b42d46_isr_855822_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/8623eb92-f4fd-4c1f-a6cd-b27ee2b42d46_isr_855822_21.stderr
parsl.dataflow.dflow DEBUG: Task 874 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 874 submitted for App characterizeImage, waiting on task 873
parsl.dataflow.dflow DEBUG: Task 874 set to pending state with AppFuture: <AppFuture at 0x14aa156fbcd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 874 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 875 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 875 submitted for App calibrate, waiting on task 874
parsl.dataflow.dflow DEBUG: Task 875 set to pending state with AppFuture: <AppFuture at 0x14aa157047d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 875 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 876 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 876 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 876 set to pending state with AppFuture: <AppFuture at 0x14aa15704d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 876 has memoization hash cfe5ee8af5cf0e6e003efba3e6f0281b
parsl.dataflow.memoization INFO: Task 876 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 876 try 0 launched on executor multi with executor id 293
parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/74499a47-312a-4ba3-9e2b-0c22d74b192d_isr_855909_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/74499a47-312a-4ba3-9e2b-0c22d74b192d_isr_855909_30.stderr
parsl.dataflow.dflow DEBUG: Task 877 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 877 submitted for App characterizeImage, waiting on task 876
parsl.dataflow.dflow DEBUG: Task 877 set to pending state with AppFuture: <AppFuture at 0x14aa15704ad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 877 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 878 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 878 submitted for App calibrate, waiting on task 877
parsl.dataflow.dflow DEBUG: Task 878 set to pending state with AppFuture: <AppFuture at 0x14aa15705810 state=pending>
parsl.dataflow.dflow DEBUG: Task 878 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 879 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 879 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 879 set to pending state with AppFuture: <AppFuture at 0x14aa15706190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 879 has memoization hash 5e055ef1c20db1b5878d899ac4f0ea55
parsl.dataflow.memoization INFO: Task 879 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156ea0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 879 try 0 launched on executor multi with executor id 294
parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a29dcab2-64fb-4816-91fb-016056abc58b_isr_855911_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a29dcab2-64fb-4816-91fb-016056abc58b_isr_855911_36.stderr
parsl.dataflow.dflow DEBUG: Task 880 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 880 submitted for App characterizeImage, waiting on task 879
parsl.dataflow.dflow DEBUG: Task 880 set to pending state with AppFuture: <AppFuture at 0x14aa157057d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 880 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 881 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 881 submitted for App calibrate, waiting on task 880
parsl.dataflow.dflow DEBUG: Task 881 set to pending state with AppFuture: <AppFuture at 0x14aa15706dd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 881 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 882 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 882 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 882 set to pending state with AppFuture: <AppFuture at 0x14aa15707610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 882 has memoization hash 193db1ae23178d74a1181659b89523b6
parsl.dataflow.memoization INFO: Task 882 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156ea3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 882 try 0 launched on executor multi with executor id 295
parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/7b31c6f5-0d7f-4c1a-8151-5afcbaedc581_isr_855910_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/7b31c6f5-0d7f-4c1a-8151-5afcbaedc581_isr_855910_38.stderr
parsl.dataflow.dflow DEBUG: Task 883 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 883 submitted for App characterizeImage, waiting on task 882
parsl.dataflow.dflow DEBUG: Task 883 set to pending state with AppFuture: <AppFuture at 0x14aa15707d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 883 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 884 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 884 submitted for App calibrate, waiting on task 883
parsl.dataflow.dflow DEBUG: Task 884 set to pending state with AppFuture: <AppFuture at 0x14aa15707b10 state=pending>
parsl.dataflow.dflow DEBUG: Task 884 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 885 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 885 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 885 set to pending state with AppFuture: <AppFuture at 0x14aa1570ca90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 885 has memoization hash 3c6eea54d4616596b5ec2ba6c0616f8e
parsl.dataflow.memoization INFO: Task 885 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156ea700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 885 try 0 launched on executor multi with executor id 296
parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/aaba556b-2031-41af-855e-aba7fb81e0e1_isr_855909_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/aaba556b-2031-41af-855e-aba7fb81e0e1_isr_855909_1.stderr
parsl.dataflow.dflow DEBUG: Task 886 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 886 submitted for App characterizeImage, waiting on task 885
parsl.dataflow.dflow DEBUG: Task 886 set to pending state with AppFuture: <AppFuture at 0x14aa1570d2d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 886 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 887 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 887 submitted for App calibrate, waiting on task 886
parsl.dataflow.dflow DEBUG: Task 887 set to pending state with AppFuture: <AppFuture at 0x14aa1570d110 state=pending>
parsl.dataflow.dflow DEBUG: Task 887 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 888 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 888 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 888 set to pending state with AppFuture: <AppFuture at 0x14aa1570de10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 888 has memoization hash 33d9ee338d5717b6893b5399d5872230
parsl.dataflow.memoization INFO: Task 888 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156eaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 888 try 0 launched on executor multi with executor id 297
parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/eaa53c28-6c30-48f4-9a87-ca71089d56b2_isr_855820_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/eaa53c28-6c30-48f4-9a87-ca71089d56b2_isr_855820_3.stderr
parsl.dataflow.dflow DEBUG: Task 889 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 889 submitted for App characterizeImage, waiting on task 888
parsl.dataflow.dflow DEBUG: Task 889 set to pending state with AppFuture: <AppFuture at 0x14aa1570d4d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 889 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 890 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 890 submitted for App calibrate, waiting on task 889
parsl.dataflow.dflow DEBUG: Task 890 set to pending state with AppFuture: <AppFuture at 0x14aa1570e8d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 890 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 891 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 891 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 891 set to pending state with AppFuture: <AppFuture at 0x14aa1570f290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 891 has memoization hash c2934d1dc98920086c05d831d2b95245
parsl.dataflow.memoization INFO: Task 891 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156ead40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 891 try 0 launched on executor multi with executor id 298
parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/d45c8313-e5cb-48c7-9e8c-197d2464c024_isr_855821_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/d45c8313-e5cb-48c7-9e8c-197d2464c024_isr_855821_40.stderr
parsl.dataflow.dflow DEBUG: Task 892 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 892 submitted for App characterizeImage, waiting on task 891
parsl.dataflow.dflow DEBUG: Task 892 set to pending state with AppFuture: <AppFuture at 0x14aa1570e750 state=pending>
parsl.dataflow.dflow DEBUG: Task 892 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 893 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 893 submitted for App calibrate, waiting on task 892
parsl.dataflow.dflow DEBUG: Task 893 set to pending state with AppFuture: <AppFuture at 0x14aa1570fa90 state=pending>
parsl.dataflow.dflow DEBUG: Task 893 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 894 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 894 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 894 set to pending state with AppFuture: <AppFuture at 0x14aa1571c6d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 894 has memoization hash 9529aa8dcf3cd3e8b5b54251d9cf4f46
parsl.dataflow.memoization INFO: Task 894 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156eb060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 894 try 0 launched on executor multi with executor id 299
parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/5bd3a889-2528-41ba-8d35-e5c0411628ab_isr_855820_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/5bd3a889-2528-41ba-8d35-e5c0411628ab_isr_855820_60.stderr
parsl.dataflow.dflow DEBUG: Task 895 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 895 submitted for App characterizeImage, waiting on task 894
parsl.dataflow.dflow DEBUG: Task 895 set to pending state with AppFuture: <AppFuture at 0x14aa1571d710 state=pending>
parsl.dataflow.dflow DEBUG: Task 895 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 896 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 896 submitted for App calibrate, waiting on task 895
parsl.dataflow.dflow DEBUG: Task 896 set to pending state with AppFuture: <AppFuture at 0x14aa1571d010 state=pending>
parsl.dataflow.dflow DEBUG: Task 896 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 897 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 897 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 897 set to pending state with AppFuture: <AppFuture at 0x14aa1571dad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 897 has memoization hash c1bcbd075144e6786070da979c68ef45
parsl.dataflow.memoization INFO: Task 897 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156eb380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 897 try 0 launched on executor multi with executor id 300
parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/6c4fb8de-1280-46e5-834f-f7e270fc8754_isr_855910_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/6c4fb8de-1280-46e5-834f-f7e270fc8754_isr_855910_43.stderr
parsl.dataflow.dflow DEBUG: Task 898 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 898 submitted for App characterizeImage, waiting on task 897
parsl.dataflow.dflow DEBUG: Task 898 set to pending state with AppFuture: <AppFuture at 0x14aa1571dc10 state=pending>
parsl.dataflow.dflow DEBUG: Task 898 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 899 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 899 submitted for App calibrate, waiting on task 898
parsl.dataflow.dflow DEBUG: Task 899 set to pending state with AppFuture: <AppFuture at 0x14aa1571e750 state=pending>
parsl.dataflow.dflow DEBUG: Task 899 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 900 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 900 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 900 set to pending state with AppFuture: <AppFuture at 0x14aa1571eed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 900 has memoization hash 18697ba6752754e09ddc11e6aa9a3e48
parsl.dataflow.memoization INFO: Task 900 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156eb6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 900 try 0 launched on executor multi with executor id 301
parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/ef0fc65c-19eb-4763-bec2-edd2c9bdf346_isr_855821_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/ef0fc65c-19eb-4763-bec2-edd2c9bdf346_isr_855821_39.stderr
parsl.dataflow.dflow DEBUG: Task 901 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 901 submitted for App characterizeImage, waiting on task 900
parsl.dataflow.dflow DEBUG: Task 901 set to pending state with AppFuture: <AppFuture at 0x14aa1571ec10 state=pending>
parsl.dataflow.dflow DEBUG: Task 901 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 902 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 902 submitted for App calibrate, waiting on task 901
parsl.dataflow.dflow DEBUG: Task 902 set to pending state with AppFuture: <AppFuture at 0x14aa1571f7d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 902 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 903 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 903 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 903 set to pending state with AppFuture: <AppFuture at 0x14aa15724310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 903 has memoization hash b2552f9d1fcacd747727e886e334f1c9
parsl.dataflow.memoization INFO: Task 903 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156eb9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 903 try 0 launched on executor multi with executor id 302
parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/fd2c77ba-9fc3-44cf-ae64-d70c1c767fe9_isr_855822_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/fd2c77ba-9fc3-44cf-ae64-d70c1c767fe9_isr_855822_61.stderr
parsl.dataflow.dflow DEBUG: Task 904 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 904 submitted for App characterizeImage, waiting on task 903
parsl.dataflow.dflow DEBUG: Task 904 set to pending state with AppFuture: <AppFuture at 0x14aa158d7690 state=pending>
parsl.dataflow.dflow DEBUG: Task 904 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 905 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 905 submitted for App calibrate, waiting on task 904
parsl.dataflow.dflow DEBUG: Task 905 set to pending state with AppFuture: <AppFuture at 0x14aa15725210 state=pending>
parsl.dataflow.dflow DEBUG: Task 905 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 906 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 906 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 906 set to pending state with AppFuture: <AppFuture at 0x14aa15725710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 906 has memoization hash f5fe4785191b5decb55f4ab7e3e14465
parsl.dataflow.memoization INFO: Task 906 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156ebce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 906 try 0 launched on executor multi with executor id 303
parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/9258bda4-91e7-4758-bd5b-596e6e9debbd_isr_855910_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/9258bda4-91e7-4758-bd5b-596e6e9debbd_isr_855910_54.stderr
parsl.dataflow.dflow DEBUG: Task 907 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 907 submitted for App characterizeImage, waiting on task 906
parsl.dataflow.dflow DEBUG: Task 907 set to pending state with AppFuture: <AppFuture at 0x14aa15725890 state=pending>
parsl.dataflow.dflow DEBUG: Task 907 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 908 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 908 submitted for App calibrate, waiting on task 907
parsl.dataflow.dflow DEBUG: Task 908 set to pending state with AppFuture: <AppFuture at 0x14aa157265d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 908 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 909 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 909 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 909 set to pending state with AppFuture: <AppFuture at 0x14aa15726ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 909 has memoization hash 6dfbcf76c03105e72af5647c401a04a2
parsl.dataflow.memoization INFO: Task 909 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15728040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 909 try 0 launched on executor multi with executor id 304
parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0c847ae1-3ae3-44f0-a563-0c12e033b723_isr_855820_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0c847ae1-3ae3-44f0-a563-0c12e033b723_isr_855820_34.stderr
parsl.dataflow.dflow DEBUG: Task 910 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 910 submitted for App characterizeImage, waiting on task 909
parsl.dataflow.dflow DEBUG: Task 910 set to pending state with AppFuture: <AppFuture at 0x14aa15726810 state=pending>
parsl.dataflow.dflow DEBUG: Task 910 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 911 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 911 submitted for App calibrate, waiting on task 910
parsl.dataflow.dflow DEBUG: Task 911 set to pending state with AppFuture: <AppFuture at 0x14aa15727210 state=pending>
parsl.dataflow.dflow DEBUG: Task 911 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 912 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 912 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 912 set to pending state with AppFuture: <AppFuture at 0x14aa15727f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 912 has memoization hash f98c71705f0112753c35c17f6ee8a55c
parsl.dataflow.memoization INFO: Task 912 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15728360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 912 try 0 launched on executor multi with executor id 305
parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/461db0e7-15e7-46fc-aeb4-c828aef366b6_isr_855910_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/461db0e7-15e7-46fc-aeb4-c828aef366b6_isr_855910_51.stderr
parsl.dataflow.dflow DEBUG: Task 913 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 913 submitted for App characterizeImage, waiting on task 912
parsl.dataflow.dflow DEBUG: Task 913 set to pending state with AppFuture: <AppFuture at 0x14aa15727cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 913 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 914 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 914 submitted for App calibrate, waiting on task 913
parsl.dataflow.dflow DEBUG: Task 914 set to pending state with AppFuture: <AppFuture at 0x14aa1572c810 state=pending>
parsl.dataflow.dflow DEBUG: Task 914 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 915 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 915 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 915 set to pending state with AppFuture: <AppFuture at 0x14aa1572d3d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 915 has memoization hash fdee21e8d9eb42edea28612b3b193fc1
parsl.dataflow.memoization INFO: Task 915 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15728680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 915 try 0 launched on executor multi with executor id 306
parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/09d543f0-8abf-497b-8e0c-050b9b945ace_isr_855910_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/09d543f0-8abf-497b-8e0c-050b9b945ace_isr_855910_6.stderr
parsl.dataflow.dflow DEBUG: Task 916 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 916 submitted for App characterizeImage, waiting on task 915
parsl.dataflow.dflow DEBUG: Task 916 set to pending state with AppFuture: <AppFuture at 0x14aa1572d510 state=pending>
parsl.dataflow.dflow DEBUG: Task 916 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 917 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 917 submitted for App calibrate, waiting on task 916
parsl.dataflow.dflow DEBUG: Task 917 set to pending state with AppFuture: <AppFuture at 0x14aa1572de90 state=pending>
parsl.dataflow.dflow DEBUG: Task 917 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 918 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 918 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 918 set to pending state with AppFuture: <AppFuture at 0x14aa1572e810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 918 has memoization hash e48c9ee7b2f122972f87e5201f762ae5
parsl.dataflow.memoization INFO: Task 918 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157289a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 918 try 0 launched on executor multi with executor id 307
parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/b73491e0-ebc1-40ac-a84d-7aded2e6b91e_isr_855822_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/b73491e0-ebc1-40ac-a84d-7aded2e6b91e_isr_855822_52.stderr
parsl.dataflow.dflow DEBUG: Task 919 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 919 submitted for App characterizeImage, waiting on task 918
parsl.dataflow.dflow DEBUG: Task 919 set to pending state with AppFuture: <AppFuture at 0x14aa1572dc10 state=pending>
parsl.dataflow.dflow DEBUG: Task 919 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 920 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 920 submitted for App calibrate, waiting on task 919
parsl.dataflow.dflow DEBUG: Task 920 set to pending state with AppFuture: <AppFuture at 0x14aa1572f090 state=pending>
parsl.dataflow.dflow DEBUG: Task 920 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 921 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 921 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 921 set to pending state with AppFuture: <AppFuture at 0x14aa1572fc10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 921 has memoization hash 862078a634d1fd876ddddd6f2c27e566
parsl.dataflow.memoization INFO: Task 921 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15728cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 921 try 0 launched on executor multi with executor id 308
parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/36cfd077-2fdc-4556-ad6d-a75a70d623ef_isr_855910_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/36cfd077-2fdc-4556-ad6d-a75a70d623ef_isr_855910_50.stderr
parsl.dataflow.dflow DEBUG: Task 922 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 922 submitted for App characterizeImage, waiting on task 921
parsl.dataflow.dflow DEBUG: Task 922 set to pending state with AppFuture: <AppFuture at 0x14aa1572ff50 state=pending>
parsl.dataflow.dflow DEBUG: Task 922 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 923 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 923 submitted for App calibrate, waiting on task 922
parsl.dataflow.dflow DEBUG: Task 923 set to pending state with AppFuture: <AppFuture at 0x14aa1573c410 state=pending>
parsl.dataflow.dflow DEBUG: Task 923 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 924 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 924 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 924 set to pending state with AppFuture: <AppFuture at 0x14aa1573d010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 924 has memoization hash 955fa11b8aed6ef81155f26c91d32ec9
parsl.dataflow.memoization INFO: Task 924 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15728fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 924 try 0 launched on executor multi with executor id 309
parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/47124ece-5fd4-4a43-9728-d71ff8eb5486_isr_855821_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/47124ece-5fd4-4a43-9728-d71ff8eb5486_isr_855821_60.stderr
parsl.dataflow.dflow DEBUG: Task 925 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 925 submitted for App characterizeImage, waiting on task 924
parsl.dataflow.dflow DEBUG: Task 925 set to pending state with AppFuture: <AppFuture at 0x14aa1573d6d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 925 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 926 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 926 submitted for App calibrate, waiting on task 925
parsl.dataflow.dflow DEBUG: Task 926 set to pending state with AppFuture: <AppFuture at 0x14aa1573cc90 state=pending>
parsl.dataflow.dflow DEBUG: Task 926 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 927 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 927 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 927 set to pending state with AppFuture: <AppFuture at 0x14aa1573e450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 927 has memoization hash 8c16590766823ff836277edd3ba4e90f
parsl.dataflow.memoization INFO: Task 927 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15729300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 927 try 0 launched on executor multi with executor id 310
parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/2a8e4445-7b0d-477d-a1c0-99200431e2bf_isr_855911_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/2a8e4445-7b0d-477d-a1c0-99200431e2bf_isr_855911_5.stderr
parsl.dataflow.dflow DEBUG: Task 928 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 928 submitted for App characterizeImage, waiting on task 927
parsl.dataflow.dflow DEBUG: Task 928 set to pending state with AppFuture: <AppFuture at 0x14aa1573d810 state=pending>
parsl.dataflow.dflow DEBUG: Task 928 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 929 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 929 submitted for App calibrate, waiting on task 928
parsl.dataflow.dflow DEBUG: Task 929 set to pending state with AppFuture: <AppFuture at 0x14aa1573eed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 929 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 930 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 930 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 930 set to pending state with AppFuture: <AppFuture at 0x14aa1573f8d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 930 has memoization hash 1863bb1d4148036062e8016de42b88e3
parsl.dataflow.memoization INFO: Task 930 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15729620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 930 try 0 launched on executor multi with executor id 311
parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/486b8cd6-1343-44c2-bc63-789071c5f890_isr_855909_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/486b8cd6-1343-44c2-bc63-789071c5f890_isr_855909_32.stderr
parsl.dataflow.dflow DEBUG: Task 931 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 931 submitted for App characterizeImage, waiting on task 930
parsl.dataflow.dflow DEBUG: Task 931 set to pending state with AppFuture: <AppFuture at 0x14aa1573fc50 state=pending>
parsl.dataflow.dflow DEBUG: Task 931 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 932 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 932 submitted for App calibrate, waiting on task 931
parsl.dataflow.dflow DEBUG: Task 932 set to pending state with AppFuture: <AppFuture at 0x14aa1574c650 state=pending>
parsl.dataflow.dflow DEBUG: Task 932 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 933 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 933 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 933 set to pending state with AppFuture: <AppFuture at 0x14aa1574cc90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 933 has memoization hash 16626109032b9c83f576662beea05b34
parsl.dataflow.memoization INFO: Task 933 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15729940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 933 try 0 launched on executor multi with executor id 312
parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/58c5d8f3-d992-4fa2-87e9-279d413f8193_isr_855821_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/58c5d8f3-d992-4fa2-87e9-279d413f8193_isr_855821_4.stderr
parsl.dataflow.dflow DEBUG: Task 934 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 934 submitted for App characterizeImage, waiting on task 933
parsl.dataflow.dflow DEBUG: Task 934 set to pending state with AppFuture: <AppFuture at 0x14aa1574c250 state=pending>
parsl.dataflow.dflow DEBUG: Task 934 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 935 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 935 submitted for App calibrate, waiting on task 934
parsl.dataflow.dflow DEBUG: Task 935 set to pending state with AppFuture: <AppFuture at 0x14aa1574d690 state=pending>
parsl.dataflow.dflow DEBUG: Task 935 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 936 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 936 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 936 set to pending state with AppFuture: <AppFuture at 0x14aa1574e050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 936 has memoization hash 41f1e349aaf37a92f139563f5a114a45
parsl.dataflow.memoization INFO: Task 936 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15729c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 936 try 0 launched on executor multi with executor id 313
parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/1653afb0-f1f6-4d4e-acf7-eabfa30d7448_isr_855822_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/1653afb0-f1f6-4d4e-acf7-eabfa30d7448_isr_855822_24.stderr
parsl.dataflow.dflow DEBUG: Task 937 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 937 submitted for App characterizeImage, waiting on task 936
parsl.dataflow.dflow DEBUG: Task 937 set to pending state with AppFuture: <AppFuture at 0x14aa1574d450 state=pending>
parsl.dataflow.dflow DEBUG: Task 937 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 938 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 938 submitted for App calibrate, waiting on task 937
parsl.dataflow.dflow DEBUG: Task 938 set to pending state with AppFuture: <AppFuture at 0x14aa1574e950 state=pending>
parsl.dataflow.dflow DEBUG: Task 938 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 939 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 939 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 939 set to pending state with AppFuture: <AppFuture at 0x14aa1574f510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 939 has memoization hash 5ce01f1c8d409de1d8882d759cb37d6a
parsl.dataflow.memoization INFO: Task 939 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15729f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 939 try 0 launched on executor multi with executor id 314
parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/0323c06f-e0ca-49a5-aa73-f9b70dd99997_isr_855911_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/0323c06f-e0ca-49a5-aa73-f9b70dd99997_isr_855911_47.stderr
parsl.dataflow.dflow DEBUG: Task 940 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 940 submitted for App characterizeImage, waiting on task 939
parsl.dataflow.dflow DEBUG: Task 940 set to pending state with AppFuture: <AppFuture at 0x14aa1574fd90 state=pending>
parsl.dataflow.dflow DEBUG: Task 940 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 941 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 941 submitted for App calibrate, waiting on task 940
parsl.dataflow.dflow DEBUG: Task 941 set to pending state with AppFuture: <AppFuture at 0x14aa1574fe50 state=pending>
parsl.dataflow.dflow DEBUG: Task 941 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 942 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 942 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 942 set to pending state with AppFuture: <AppFuture at 0x14aa15754950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 942 has memoization hash 7cc58aea66a0f81881becbb73b2a94be
parsl.dataflow.memoization INFO: Task 942 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 942 try 0 launched on executor multi with executor id 315
parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/46766e17-037c-4a45-a6ff-7a4205663c99_isr_855911_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/46766e17-037c-4a45-a6ff-7a4205663c99_isr_855911_8.stderr
parsl.dataflow.dflow DEBUG: Task 943 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 943 submitted for App characterizeImage, waiting on task 942
parsl.dataflow.dflow DEBUG: Task 943 set to pending state with AppFuture: <AppFuture at 0x14aa15755050 state=pending>
parsl.dataflow.dflow DEBUG: Task 943 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 944 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 944 submitted for App calibrate, waiting on task 943
parsl.dataflow.dflow DEBUG: Task 944 set to pending state with AppFuture: <AppFuture at 0x14aa15755890 state=pending>
parsl.dataflow.dflow DEBUG: Task 944 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 945 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 945 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 945 set to pending state with AppFuture: <AppFuture at 0x14aa15755d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 945 has memoization hash a17ac253ed58097e0f02b3f9ef8fdf7c
parsl.dataflow.memoization INFO: Task 945 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 945 try 0 launched on executor multi with executor id 316
parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/0566f0cf-f47d-4089-a93a-19ef90867295_isr_855822_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/0566f0cf-f47d-4089-a93a-19ef90867295_isr_855822_44.stderr
parsl.dataflow.dflow DEBUG: Task 946 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 946 submitted for App characterizeImage, waiting on task 945
parsl.dataflow.dflow DEBUG: Task 946 set to pending state with AppFuture: <AppFuture at 0x14aa15756390 state=pending>
parsl.dataflow.dflow DEBUG: Task 946 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 947 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 947 submitted for App calibrate, waiting on task 946
parsl.dataflow.dflow DEBUG: Task 947 set to pending state with AppFuture: <AppFuture at 0x14aa157569d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 947 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 948 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 948 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 948 set to pending state with AppFuture: <AppFuture at 0x14aa15757110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 948 has memoization hash 58f9e27ef1d04b9bfa39d0a7696a4094
parsl.dataflow.memoization INFO: Task 948 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 948 try 0 launched on executor multi with executor id 317
parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/cab0bc88-87a2-4c85-a46b-ad08242cca8e_isr_855821_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/cab0bc88-87a2-4c85-a46b-ad08242cca8e_isr_855821_50.stderr
parsl.dataflow.dflow DEBUG: Task 949 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 949 submitted for App characterizeImage, waiting on task 948
parsl.dataflow.dflow DEBUG: Task 949 set to pending state with AppFuture: <AppFuture at 0x14aa15757d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 949 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 950 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 950 submitted for App calibrate, waiting on task 949
parsl.dataflow.dflow DEBUG: Task 950 set to pending state with AppFuture: <AppFuture at 0x14aa1570f6d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 950 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 951 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 951 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 951 set to pending state with AppFuture: <AppFuture at 0x14aa15758510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 951 has memoization hash 1703c9379a36b220ca51b2e9b2f43aa4
parsl.dataflow.memoization INFO: Task 951 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 951 try 0 launched on executor multi with executor id 318
parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/6bd1bff9-1fff-4aa8-a44c-7b29dcfac192_isr_855910_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/6bd1bff9-1fff-4aa8-a44c-7b29dcfac192_isr_855910_29.stderr
parsl.dataflow.dflow DEBUG: Task 952 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 952 submitted for App characterizeImage, waiting on task 951
parsl.dataflow.dflow DEBUG: Task 952 set to pending state with AppFuture: <AppFuture at 0x14aa15759550 state=pending>
parsl.dataflow.dflow DEBUG: Task 952 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 953 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 953 submitted for App calibrate, waiting on task 952
parsl.dataflow.dflow DEBUG: Task 953 set to pending state with AppFuture: <AppFuture at 0x14aa15759490 state=pending>
parsl.dataflow.dflow DEBUG: Task 953 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 954 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 954 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 954 set to pending state with AppFuture: <AppFuture at 0x14aa15759990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 954 has memoization hash 5b5649b4c0a5093dbd005c8cd05b2a71
parsl.dataflow.memoization INFO: Task 954 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 954 try 0 launched on executor multi with executor id 319
parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/feb42d1c-a7b6-43c0-b170-31da5d4e70ae_isr_855909_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/feb42d1c-a7b6-43c0-b170-31da5d4e70ae_isr_855909_10.stderr
parsl.dataflow.dflow DEBUG: Task 955 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 955 submitted for App characterizeImage, waiting on task 954
parsl.dataflow.dflow DEBUG: Task 955 set to pending state with AppFuture: <AppFuture at 0x14aa1575a210 state=pending>
parsl.dataflow.dflow DEBUG: Task 955 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 956 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 956 submitted for App calibrate, waiting on task 955
parsl.dataflow.dflow DEBUG: Task 956 set to pending state with AppFuture: <AppFuture at 0x14aa1575a1d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 956 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 957 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 957 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 957 set to pending state with AppFuture: <AppFuture at 0x14aa1575add0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 957 has memoization hash 9566d2ec235d10975fb28391ae4af4c4
parsl.dataflow.memoization INFO: Task 957 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 957 try 0 launched on executor multi with executor id 320
parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/3bf26ae9-c5fc-4364-96a3-dc8539da8bc8_isr_855910_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/3bf26ae9-c5fc-4364-96a3-dc8539da8bc8_isr_855910_61.stderr
parsl.dataflow.dflow DEBUG: Task 958 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 958 submitted for App characterizeImage, waiting on task 957
parsl.dataflow.dflow DEBUG: Task 958 set to pending state with AppFuture: <AppFuture at 0x14aa1575ab10 state=pending>
parsl.dataflow.dflow DEBUG: Task 958 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 959 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 959 submitted for App calibrate, waiting on task 958
parsl.dataflow.dflow DEBUG: Task 959 set to pending state with AppFuture: <AppFuture at 0x14aa1575b610 state=pending>
parsl.dataflow.dflow DEBUG: Task 959 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 960 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 960 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 960 set to pending state with AppFuture: <AppFuture at 0x14aa15768110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 960 has memoization hash d624d200f779d21cb3f19cced9c4f74b
parsl.dataflow.memoization INFO: Task 960 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 960 try 0 launched on executor multi with executor id 321
parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/59f7c88d-61c5-405e-8a3b-0a6fcc9cebf3_isr_855909_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/59f7c88d-61c5-405e-8a3b-0a6fcc9cebf3_isr_855909_42.stderr
parsl.dataflow.dflow DEBUG: Task 961 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 961 submitted for App characterizeImage, waiting on task 960
parsl.dataflow.dflow DEBUG: Task 961 set to pending state with AppFuture: <AppFuture at 0x14aa15758e50 state=pending>
parsl.dataflow.dflow DEBUG: Task 961 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 962 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 962 submitted for App calibrate, waiting on task 961
parsl.dataflow.dflow DEBUG: Task 962 set to pending state with AppFuture: <AppFuture at 0x14aa15768ad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 962 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 963 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 963 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 963 set to pending state with AppFuture: <AppFuture at 0x14aa15769610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 963 has memoization hash 81dee5fd781028e5f34486ff2b02b979
parsl.dataflow.memoization INFO: Task 963 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 963 try 0 launched on executor multi with executor id 322
parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/8ba65ba4-e46a-4444-bfe2-bbbb53c9dcbd_isr_855909_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/8ba65ba4-e46a-4444-bfe2-bbbb53c9dcbd_isr_855909_40.stderr
parsl.dataflow.dflow DEBUG: Task 964 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 964 submitted for App characterizeImage, waiting on task 963
parsl.dataflow.dflow DEBUG: Task 964 set to pending state with AppFuture: <AppFuture at 0x14aa15769d50 state=pending>
parsl.dataflow.dflow DEBUG: Task 964 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 965 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 965 submitted for App calibrate, waiting on task 964
parsl.dataflow.dflow DEBUG: Task 965 set to pending state with AppFuture: <AppFuture at 0x14aa1576a010 state=pending>
parsl.dataflow.dflow DEBUG: Task 965 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 966 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 966 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 966 set to pending state with AppFuture: <AppFuture at 0x14aa1576aad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 966 has memoization hash a3480e74e074fc1db4808d47f23e8859
parsl.dataflow.memoization INFO: Task 966 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 966 try 0 launched on executor multi with executor id 323
parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/8a2ccb3c-c55f-430d-a07e-9da2b3d5d67a_isr_855822_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/8a2ccb3c-c55f-430d-a07e-9da2b3d5d67a_isr_855822_33.stderr
parsl.dataflow.dflow DEBUG: Task 967 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 967 submitted for App characterizeImage, waiting on task 966
parsl.dataflow.dflow DEBUG: Task 967 set to pending state with AppFuture: <AppFuture at 0x14aa15769dd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 967 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 968 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 968 submitted for App calibrate, waiting on task 967
parsl.dataflow.dflow DEBUG: Task 968 set to pending state with AppFuture: <AppFuture at 0x14aa1576b7d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 968 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 969 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 969 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 969 set to pending state with AppFuture: <AppFuture at 0x14aa1576bf10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 969 has memoization hash 18d45166b33588a54a3a26eea2d282a5
parsl.dataflow.memoization INFO: Task 969 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 969 try 0 launched on executor multi with executor id 324
parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/5f3e9e52-4c84-40ce-8b34-64154b79787e_isr_855911_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/5f3e9e52-4c84-40ce-8b34-64154b79787e_isr_855911_48.stderr
parsl.dataflow.dflow DEBUG: Task 970 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 970 submitted for App characterizeImage, waiting on task 969
parsl.dataflow.dflow DEBUG: Task 970 set to pending state with AppFuture: <AppFuture at 0x14aa1576bc50 state=pending>
parsl.dataflow.dflow DEBUG: Task 970 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 971 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 971 submitted for App calibrate, waiting on task 970
parsl.dataflow.dflow DEBUG: Task 971 set to pending state with AppFuture: <AppFuture at 0x14aa15778910 state=pending>
parsl.dataflow.dflow DEBUG: Task 971 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 972 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 972 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 972 set to pending state with AppFuture: <AppFuture at 0x14aa15779290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 972 has memoization hash 077cacdf3fb1c0581b3e9e46a87aef50
parsl.dataflow.memoization INFO: Task 972 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 972 try 0 launched on executor multi with executor id 325
parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/8e3addf9-7afa-4268-a9d2-a609ae9c4ba2_isr_855821_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/8e3addf9-7afa-4268-a9d2-a609ae9c4ba2_isr_855821_5.stderr
parsl.dataflow.dflow DEBUG: Task 973 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 973 submitted for App characterizeImage, waiting on task 972
parsl.dataflow.dflow DEBUG: Task 973 set to pending state with AppFuture: <AppFuture at 0x14aa15778fd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 973 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 974 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 974 submitted for App calibrate, waiting on task 973
parsl.dataflow.dflow DEBUG: Task 974 set to pending state with AppFuture: <AppFuture at 0x14aa15779c50 state=pending>
parsl.dataflow.dflow DEBUG: Task 974 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 975 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 975 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 975 set to pending state with AppFuture: <AppFuture at 0x14aa1577a6d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 975 has memoization hash 31639962eec0682c736c58c450cdd56c
parsl.dataflow.memoization INFO: Task 975 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 975 try 0 launched on executor multi with executor id 326
parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/e3c10186-a3da-403f-889f-7f17475d64d7_isr_855820_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/e3c10186-a3da-403f-889f-7f17475d64d7_isr_855820_46.stderr
parsl.dataflow.dflow DEBUG: Task 976 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 976 submitted for App characterizeImage, waiting on task 975
parsl.dataflow.dflow DEBUG: Task 976 set to pending state with AppFuture: <AppFuture at 0x14aa1577b510 state=pending>
parsl.dataflow.dflow DEBUG: Task 976 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 977 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 977 submitted for App calibrate, waiting on task 976
parsl.dataflow.dflow DEBUG: Task 977 set to pending state with AppFuture: <AppFuture at 0x14aa1577b4d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 977 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 978 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 978 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 978 set to pending state with AppFuture: <AppFuture at 0x14aa1577bad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 978 has memoization hash 476e3361f7373bea6265c1feb709ab4d
parsl.dataflow.memoization INFO: Task 978 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 978 try 0 launched on executor multi with executor id 327
parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/af20380d-5e9f-413e-8cad-812800e6f4f5_isr_855911_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/af20380d-5e9f-413e-8cad-812800e6f4f5_isr_855911_27.stderr
parsl.dataflow.dflow DEBUG: Task 979 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 979 submitted for App characterizeImage, waiting on task 978
parsl.dataflow.dflow DEBUG: Task 979 set to pending state with AppFuture: <AppFuture at 0x14aa1577bed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 979 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 980 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 980 submitted for App calibrate, waiting on task 979
parsl.dataflow.dflow DEBUG: Task 980 set to pending state with AppFuture: <AppFuture at 0x14aa15788990 state=pending>
parsl.dataflow.dflow DEBUG: Task 980 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 981 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 981 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 981 set to pending state with AppFuture: <AppFuture at 0x14aa15788ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 981 has memoization hash be55ec840693439c10968da22a878891
parsl.dataflow.memoization INFO: Task 981 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 981 try 0 launched on executor multi with executor id 328
parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a716fc96-5512-4781-baa0-77afd728b2df_isr_855911_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a716fc96-5512-4781-baa0-77afd728b2df_isr_855911_30.stderr
parsl.dataflow.dflow DEBUG: Task 982 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 982 submitted for App characterizeImage, waiting on task 981
parsl.dataflow.dflow DEBUG: Task 982 set to pending state with AppFuture: <AppFuture at 0x14aa15881490 state=pending>
parsl.dataflow.dflow DEBUG: Task 982 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 983 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 983 submitted for App calibrate, waiting on task 982
parsl.dataflow.dflow DEBUG: Task 983 set to pending state with AppFuture: <AppFuture at 0x14aa15789790 state=pending>
parsl.dataflow.dflow DEBUG: Task 983 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 984 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 984 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 984 set to pending state with AppFuture: <AppFuture at 0x14aa1578a2d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 984 has memoization hash 34c889da93b210397edef2771c57b605
parsl.dataflow.memoization INFO: Task 984 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 984 try 0 launched on executor multi with executor id 329
parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/1dce9fe6-80ee-4e34-a8d6-5cd1d4cacd6a_isr_855911_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/1dce9fe6-80ee-4e34-a8d6-5cd1d4cacd6a_isr_855911_58.stderr
parsl.dataflow.dflow DEBUG: Task 985 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 985 submitted for App characterizeImage, waiting on task 984
parsl.dataflow.dflow DEBUG: Task 985 set to pending state with AppFuture: <AppFuture at 0x14aa157891d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 985 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 986 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 986 submitted for App calibrate, waiting on task 985
parsl.dataflow.dflow DEBUG: Task 986 set to pending state with AppFuture: <AppFuture at 0x14aa1578b350 state=pending>
parsl.dataflow.dflow DEBUG: Task 986 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 987 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 987 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 987 set to pending state with AppFuture: <AppFuture at 0x14aa1578b750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 987 has memoization hash 2a4cc6b0c3ea0c1dba520ddd3199bb7a
parsl.dataflow.memoization INFO: Task 987 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 987 try 0 launched on executor multi with executor id 330
parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/ab28f426-ca4c-4c9e-9300-a7cac91eb9ff_isr_855820_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/ab28f426-ca4c-4c9e-9300-a7cac91eb9ff_isr_855820_21.stderr
parsl.dataflow.dflow DEBUG: Task 988 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 988 submitted for App characterizeImage, waiting on task 987
parsl.dataflow.dflow DEBUG: Task 988 set to pending state with AppFuture: <AppFuture at 0x14aa15788510 state=pending>
parsl.dataflow.dflow DEBUG: Task 988 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 989 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 989 submitted for App calibrate, waiting on task 988
parsl.dataflow.dflow DEBUG: Task 989 set to pending state with AppFuture: <AppFuture at 0x14aa15794550 state=pending>
parsl.dataflow.dflow DEBUG: Task 989 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 990 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 990 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 990 set to pending state with AppFuture: <AppFuture at 0x14aa15794bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 990 has memoization hash ecf2b345c827f086048f974067b0c4d5
parsl.dataflow.memoization INFO: Task 990 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 990 try 0 launched on executor multi with executor id 331
parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/232a0318-6b06-4300-92fd-a9165546e719_isr_855822_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/232a0318-6b06-4300-92fd-a9165546e719_isr_855822_34.stderr
parsl.dataflow.dflow DEBUG: Task 991 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 991 submitted for App characterizeImage, waiting on task 990
parsl.dataflow.dflow DEBUG: Task 991 set to pending state with AppFuture: <AppFuture at 0x14aa15705bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 991 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 992 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 992 submitted for App calibrate, waiting on task 991
parsl.dataflow.dflow DEBUG: Task 992 set to pending state with AppFuture: <AppFuture at 0x14aa15795390 state=pending>
parsl.dataflow.dflow DEBUG: Task 992 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 993 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 993 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 993 set to pending state with AppFuture: <AppFuture at 0x14aa15795f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 993 has memoization hash 9a7861470c4f7969398bf6753dfacf13
parsl.dataflow.memoization INFO: Task 993 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 993 try 0 launched on executor multi with executor id 332
parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/f64bebbb-f618-490b-a72b-365abe74fe95_isr_855910_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/f64bebbb-f618-490b-a72b-365abe74fe95_isr_855910_9.stderr
parsl.dataflow.dflow DEBUG: Task 994 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 994 submitted for App characterizeImage, waiting on task 993
parsl.dataflow.dflow DEBUG: Task 994 set to pending state with AppFuture: <AppFuture at 0x14aa15796110 state=pending>
parsl.dataflow.dflow DEBUG: Task 994 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 995 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 995 submitted for App calibrate, waiting on task 994
parsl.dataflow.dflow DEBUG: Task 995 set to pending state with AppFuture: <AppFuture at 0x14aa15796790 state=pending>
parsl.dataflow.dflow DEBUG: Task 995 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 996 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 996 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 996 set to pending state with AppFuture: <AppFuture at 0x14aa15797350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 996 has memoization hash 80ba1f86678d4d8d2df15c42282d6ee1
parsl.dataflow.memoization INFO: Task 996 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 996 try 0 launched on executor multi with executor id 333
parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/d1089740-bd29-4381-a280-c87153092af5_isr_855822_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/d1089740-bd29-4381-a280-c87153092af5_isr_855822_18.stderr
parsl.dataflow.dflow DEBUG: Task 997 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 997 submitted for App characterizeImage, waiting on task 996
parsl.dataflow.dflow DEBUG: Task 997 set to pending state with AppFuture: <AppFuture at 0x14aa15797d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 997 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 998 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 998 submitted for App calibrate, waiting on task 997
parsl.dataflow.dflow DEBUG: Task 998 set to pending state with AppFuture: <AppFuture at 0x14aa15797850 state=pending>
parsl.dataflow.dflow DEBUG: Task 998 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 999 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 999 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 999 set to pending state with AppFuture: <AppFuture at 0x14aa1579c7d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 999 has memoization hash 9585ffb3f854db08a9344efe7c95ff67
parsl.dataflow.memoization INFO: Task 999 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 999 try 0 launched on executor multi with executor id 334
parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/8044c1ac-9ea3-40dc-9148-f3006ec9c833_isr_855911_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/8044c1ac-9ea3-40dc-9148-f3006ec9c833_isr_855911_35.stderr
parsl.dataflow.dflow DEBUG: Task 1000 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1000 submitted for App characterizeImage, waiting on task 999
parsl.dataflow.dflow DEBUG: Task 1000 set to pending state with AppFuture: <AppFuture at 0x14aa157550d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1000 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1001 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1001 submitted for App calibrate, waiting on task 1000
parsl.dataflow.dflow DEBUG: Task 1001 set to pending state with AppFuture: <AppFuture at 0x14aa1579d010 state=pending>
parsl.dataflow.dflow DEBUG: Task 1001 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1002 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1002 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1002 set to pending state with AppFuture: <AppFuture at 0x14aa1579db90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1002 has memoization hash 9e888f37e5a91fcb9e4e289688d83145
parsl.dataflow.memoization INFO: Task 1002 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1002 try 0 launched on executor multi with executor id 335
parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/8443acdf-d0ab-4a16-957c-db4efb471e67_isr_855911_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/8443acdf-d0ab-4a16-957c-db4efb471e67_isr_855911_32.stderr
parsl.dataflow.dflow DEBUG: Task 1003 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1003 submitted for App characterizeImage, waiting on task 1002
parsl.dataflow.dflow DEBUG: Task 1003 set to pending state with AppFuture: <AppFuture at 0x14aa1579ebd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1003 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1004 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1004 submitted for App calibrate, waiting on task 1003
parsl.dataflow.dflow DEBUG: Task 1004 set to pending state with AppFuture: <AppFuture at 0x14aa1579e450 state=pending>
parsl.dataflow.dflow DEBUG: Task 1004 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1005 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1005 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1005 set to pending state with AppFuture: <AppFuture at 0x14aa1579ef90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1005 has memoization hash f18785e410e0a1a1d69fc895803c862b
parsl.dataflow.memoization INFO: Task 1005 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1005 try 0 launched on executor multi with executor id 336
parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/ae8f990f-7450-4571-999e-34b1c52c2b89_isr_855909_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/ae8f990f-7450-4571-999e-34b1c52c2b89_isr_855909_21.stderr
parsl.dataflow.dflow DEBUG: Task 1006 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1006 submitted for App characterizeImage, waiting on task 1005
parsl.dataflow.dflow DEBUG: Task 1006 set to pending state with AppFuture: <AppFuture at 0x14aa1579f350 state=pending>
parsl.dataflow.dflow DEBUG: Task 1006 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1007 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1007 submitted for App calibrate, waiting on task 1006
parsl.dataflow.dflow DEBUG: Task 1007 set to pending state with AppFuture: <AppFuture at 0x14aa1579f890 state=pending>
parsl.dataflow.dflow DEBUG: Task 1007 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1008 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1008 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1008 set to pending state with AppFuture: <AppFuture at 0x14aa157a0390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1008 has memoization hash d887ff5ae7fe11406491a3d1389d5aea
parsl.dataflow.memoization INFO: Task 1008 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1008 try 0 launched on executor multi with executor id 337
parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/74e71e55-77a6-4c79-8db5-e759865a3b5a_isr_855822_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/74e71e55-77a6-4c79-8db5-e759865a3b5a_isr_855822_29.stderr
parsl.dataflow.dflow DEBUG: Task 1009 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1009 submitted for App characterizeImage, waiting on task 1008
parsl.dataflow.dflow DEBUG: Task 1009 set to pending state with AppFuture: <AppFuture at 0x14aa157a0510 state=pending>
parsl.dataflow.dflow DEBUG: Task 1009 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1010 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1010 submitted for App calibrate, waiting on task 1009
parsl.dataflow.dflow DEBUG: Task 1010 set to pending state with AppFuture: <AppFuture at 0x14aa157a11d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1010 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1011 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1011 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1011 set to pending state with AppFuture: <AppFuture at 0x14aa157a1790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1011 has memoization hash 8f531f9422fa2f1ecdb2912c952fd7cf
parsl.dataflow.memoization INFO: Task 1011 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1011 try 0 launched on executor multi with executor id 338
parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/d923c619-47b8-4965-9e17-31d9609283e2_isr_855910_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/d923c619-47b8-4965-9e17-31d9609283e2_isr_855910_42.stderr
parsl.dataflow.dflow DEBUG: Task 1012 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1012 submitted for App characterizeImage, waiting on task 1011
parsl.dataflow.dflow DEBUG: Task 1012 set to pending state with AppFuture: <AppFuture at 0x14aa157a1250 state=pending>
parsl.dataflow.dflow DEBUG: Task 1012 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1013 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1013 submitted for App calibrate, waiting on task 1012
parsl.dataflow.dflow DEBUG: Task 1013 set to pending state with AppFuture: <AppFuture at 0x14aa157a1e50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1013 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1014 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1014 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1014 set to pending state with AppFuture: <AppFuture at 0x14aa157a2bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1014 has memoization hash 65a0a70986d838fc1ba0484682d6d8a0
parsl.dataflow.memoization INFO: Task 1014 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1014 try 0 launched on executor multi with executor id 339
parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/38bac66a-25d7-4d7f-a59f-ccc61dfd833d_isr_855909_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/38bac66a-25d7-4d7f-a59f-ccc61dfd833d_isr_855909_6.stderr
parsl.dataflow.dflow DEBUG: Task 1015 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1015 submitted for App characterizeImage, waiting on task 1014
parsl.dataflow.dflow DEBUG: Task 1015 set to pending state with AppFuture: <AppFuture at 0x14aa15822510 state=pending>
parsl.dataflow.dflow DEBUG: Task 1015 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1016 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1016 submitted for App calibrate, waiting on task 1015
parsl.dataflow.dflow DEBUG: Task 1016 set to pending state with AppFuture: <AppFuture at 0x14aa157a3410 state=pending>
parsl.dataflow.dflow DEBUG: Task 1016 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1017 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1017 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1017 set to pending state with AppFuture: <AppFuture at 0x14aa157a3f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1017 has memoization hash e168c1bf7e81f85fb9820487c8dbb6b3
parsl.dataflow.memoization INFO: Task 1017 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1017 try 0 launched on executor multi with executor id 340
parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/8dff2a86-baa3-4bc9-9a27-8de6a73ae0db_isr_855822_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/8dff2a86-baa3-4bc9-9a27-8de6a73ae0db_isr_855822_5.stderr
parsl.dataflow.dflow DEBUG: Task 1018 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1018 submitted for App characterizeImage, waiting on task 1017
parsl.dataflow.dflow DEBUG: Task 1018 set to pending state with AppFuture: <AppFuture at 0x14aa157a3bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1018 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1019 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1019 submitted for App calibrate, waiting on task 1018
parsl.dataflow.dflow DEBUG: Task 1019 set to pending state with AppFuture: <AppFuture at 0x14aa157b4690 state=pending>
parsl.dataflow.dflow DEBUG: Task 1019 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1020 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1020 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1020 set to pending state with AppFuture: <AppFuture at 0x14aa157b5190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1020 has memoization hash 03a3d33050f870348130b4c289dd21ff
parsl.dataflow.memoization INFO: Task 1020 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1020 try 0 launched on executor multi with executor id 341
parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/387f12b4-4060-4daa-a361-798b573c91e9_isr_855910_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/387f12b4-4060-4daa-a361-798b573c91e9_isr_855910_10.stderr
parsl.dataflow.dflow DEBUG: Task 1021 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1021 submitted for App characterizeImage, waiting on task 1020
parsl.dataflow.dflow DEBUG: Task 1021 set to pending state with AppFuture: <AppFuture at 0x14aa157b47d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1021 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1022 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1022 submitted for App calibrate, waiting on task 1021
parsl.dataflow.dflow DEBUG: Task 1022 set to pending state with AppFuture: <AppFuture at 0x14aa157b5850 state=pending>
parsl.dataflow.dflow DEBUG: Task 1022 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1023 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1023 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1023 set to pending state with AppFuture: <AppFuture at 0x14aa157b6810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1023 has memoization hash 9c88f9c5b937eb44dcb3a1f041163e07
parsl.dataflow.memoization INFO: Task 1023 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1023 try 0 launched on executor multi with executor id 342
parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/6499acfb-9467-4273-b59f-479f31254b9c_isr_855909_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/6499acfb-9467-4273-b59f-479f31254b9c_isr_855909_19.stderr
parsl.dataflow.dflow DEBUG: Task 1024 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1024 submitted for App characterizeImage, waiting on task 1023
parsl.dataflow.dflow DEBUG: Task 1024 set to pending state with AppFuture: <AppFuture at 0x14aa157b6950 state=pending>
parsl.dataflow.dflow DEBUG: Task 1024 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1025 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1025 submitted for App calibrate, waiting on task 1024
parsl.dataflow.dflow DEBUG: Task 1025 set to pending state with AppFuture: <AppFuture at 0x14aa157b7290 state=pending>
parsl.dataflow.dflow DEBUG: Task 1025 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1026 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1026 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1026 set to pending state with AppFuture: <AppFuture at 0x14aa157b7c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1026 has memoization hash 6d4390f66cb2f74ed79ac5e8dfb3abcd
parsl.dataflow.memoization INFO: Task 1026 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1026 try 0 launched on executor multi with executor id 343
parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/8a18c895-bd26-4245-9022-4cbb8518ad05_isr_855909_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/8a18c895-bd26-4245-9022-4cbb8518ad05_isr_855909_15.stderr
parsl.dataflow.dflow DEBUG: Task 1027 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1027 submitted for App characterizeImage, waiting on task 1026
parsl.dataflow.dflow DEBUG: Task 1027 set to pending state with AppFuture: <AppFuture at 0x14aa157b7f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1027 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1028 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1028 submitted for App calibrate, waiting on task 1027
parsl.dataflow.dflow DEBUG: Task 1028 set to pending state with AppFuture: <AppFuture at 0x14aa157c08d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1028 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1029 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1029 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1029 set to pending state with AppFuture: <AppFuture at 0x14aa157c1010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1029 has memoization hash 3b636fd7f2fd9ecec4b6000c3c8d2093
parsl.dataflow.memoization INFO: Task 1029 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1029 try 0 launched on executor multi with executor id 344
parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/4f3c85f1-4cde-4734-bd56-3b902f9bd5a0_isr_855821_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/4f3c85f1-4cde-4734-bd56-3b902f9bd5a0_isr_855821_19.stderr
parsl.dataflow.dflow DEBUG: Task 1030 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1030 submitted for App characterizeImage, waiting on task 1029
parsl.dataflow.dflow DEBUG: Task 1030 set to pending state with AppFuture: <AppFuture at 0x14aa157c0590 state=pending>
parsl.dataflow.dflow DEBUG: Task 1030 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1031 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1031 submitted for App calibrate, waiting on task 1030
parsl.dataflow.dflow DEBUG: Task 1031 set to pending state with AppFuture: <AppFuture at 0x14aa157c18d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1031 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1032 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1032 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1032 set to pending state with AppFuture: <AppFuture at 0x14aa157c2390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1032 has memoization hash b8174973bdcaa987881bb1942a6de8a3
parsl.dataflow.memoization INFO: Task 1032 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1032 try 0 launched on executor multi with executor id 345
parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/16dcc6f9-8e68-43b4-bfec-ad5f502989b9_isr_855822_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/16dcc6f9-8e68-43b4-bfec-ad5f502989b9_isr_855822_4.stderr
parsl.dataflow.dflow DEBUG: Task 1033 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1033 submitted for App characterizeImage, waiting on task 1032
parsl.dataflow.dflow DEBUG: Task 1033 set to pending state with AppFuture: <AppFuture at 0x14aa157c17d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1033 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1034 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1034 submitted for App calibrate, waiting on task 1033
parsl.dataflow.dflow DEBUG: Task 1034 set to pending state with AppFuture: <AppFuture at 0x14aa157c3490 state=pending>
parsl.dataflow.dflow DEBUG: Task 1034 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1035 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1035 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1035 set to pending state with AppFuture: <AppFuture at 0x14aa157c38d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1035 has memoization hash 266ed8db758f64182ec9c8c4365dc8d4
parsl.dataflow.memoization INFO: Task 1035 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1035 try 0 launched on executor multi with executor id 346
parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/a9f463ee-efcc-4cae-a92b-49aa12d1a1c6_isr_855820_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/a9f463ee-efcc-4cae-a92b-49aa12d1a1c6_isr_855820_57.stderr
parsl.dataflow.dflow DEBUG: Task 1036 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1036 submitted for App characterizeImage, waiting on task 1035
parsl.dataflow.dflow DEBUG: Task 1036 set to pending state with AppFuture: <AppFuture at 0x14aa157c2b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1036 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1037 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1037 submitted for App calibrate, waiting on task 1036
parsl.dataflow.dflow DEBUG: Task 1037 set to pending state with AppFuture: <AppFuture at 0x14aa157c88d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1037 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1038 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1038 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1038 set to pending state with AppFuture: <AppFuture at 0x14aa157c8cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1038 has memoization hash c196abc422c3bca46262ec1da744e390
parsl.dataflow.memoization INFO: Task 1038 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1038 try 0 launched on executor multi with executor id 347
parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0c9ef19c-5af6-4f0f-98e6-e686d51066e7_isr_855820_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0c9ef19c-5af6-4f0f-98e6-e686d51066e7_isr_855820_39.stderr
parsl.dataflow.dflow DEBUG: Task 1039 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1039 submitted for App characterizeImage, waiting on task 1038
parsl.dataflow.dflow DEBUG: Task 1039 set to pending state with AppFuture: <AppFuture at 0x14aa15779a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1039 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1040 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1040 submitted for App calibrate, waiting on task 1039
parsl.dataflow.dflow DEBUG: Task 1040 set to pending state with AppFuture: <AppFuture at 0x14aa157c9b10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1040 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1041 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1041 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1041 set to pending state with AppFuture: <AppFuture at 0x14aa157ca090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1041 has memoization hash 2f4a2d5055b46435cff8700cfdd2cbdb
parsl.dataflow.memoization INFO: Task 1041 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1041 try 0 launched on executor multi with executor id 348
parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/c0e73a22-8459-4a99-9a09-b91ee46a226e_isr_855911_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/c0e73a22-8459-4a99-9a09-b91ee46a226e_isr_855911_45.stderr
parsl.dataflow.dflow DEBUG: Task 1042 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1042 submitted for App characterizeImage, waiting on task 1041
parsl.dataflow.dflow DEBUG: Task 1042 set to pending state with AppFuture: <AppFuture at 0x14aa157c9dd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1042 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1043 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1043 submitted for App calibrate, waiting on task 1042
parsl.dataflow.dflow DEBUG: Task 1043 set to pending state with AppFuture: <AppFuture at 0x14aa157ca8d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1043 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1044 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1044 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1044 set to pending state with AppFuture: <AppFuture at 0x14aa157cb510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1044 has memoization hash 240fd618ca387d52dc99a8a9b78c8074
parsl.dataflow.memoization INFO: Task 1044 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1044 try 0 launched on executor multi with executor id 349
parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/85cd8538-007e-4d21-b05b-75e5b1e7c63d_isr_855820_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/85cd8538-007e-4d21-b05b-75e5b1e7c63d_isr_855820_17.stderr
parsl.dataflow.dflow DEBUG: Task 1045 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1045 submitted for App characterizeImage, waiting on task 1044
parsl.dataflow.dflow DEBUG: Task 1045 set to pending state with AppFuture: <AppFuture at 0x14aa157cb810 state=pending>
parsl.dataflow.dflow DEBUG: Task 1045 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1046 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1046 submitted for App calibrate, waiting on task 1045
parsl.dataflow.dflow DEBUG: Task 1046 set to pending state with AppFuture: <AppFuture at 0x14aa155dc250 state=pending>
parsl.dataflow.dflow DEBUG: Task 1046 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1047 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1047 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1047 set to pending state with AppFuture: <AppFuture at 0x14aa155dc990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1047 has memoization hash 87c4d511913f354618d9b61eb001c037
parsl.dataflow.memoization INFO: Task 1047 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1047 try 0 launched on executor multi with executor id 350
parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/c4835a8b-9e85-4ae3-983a-815913e97830_isr_855821_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/c4835a8b-9e85-4ae3-983a-815913e97830_isr_855821_28.stderr
parsl.dataflow.dflow DEBUG: Task 1048 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1048 submitted for App characterizeImage, waiting on task 1047
parsl.dataflow.dflow DEBUG: Task 1048 set to pending state with AppFuture: <AppFuture at 0x14aa155dd210 state=pending>
parsl.dataflow.dflow DEBUG: Task 1048 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1049 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1049 submitted for App calibrate, waiting on task 1048
parsl.dataflow.dflow DEBUG: Task 1049 set to pending state with AppFuture: <AppFuture at 0x14aa155dd110 state=pending>
parsl.dataflow.dflow DEBUG: Task 1049 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1050 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1050 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1050 set to pending state with AppFuture: <AppFuture at 0x14aa155ddd50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1050 has memoization hash 1f88418623f704b3995087d634b9e605
parsl.dataflow.memoization INFO: Task 1050 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1050 try 0 launched on executor multi with executor id 351
parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/f8d7cfed-b61d-40f1-ad5f-caa89d0c5fb7_isr_855820_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/f8d7cfed-b61d-40f1-ad5f-caa89d0c5fb7_isr_855820_53.stderr
parsl.dataflow.dflow DEBUG: Task 1051 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1051 submitted for App characterizeImage, waiting on task 1050
parsl.dataflow.dflow DEBUG: Task 1051 set to pending state with AppFuture: <AppFuture at 0x14aa15865310 state=pending>
parsl.dataflow.dflow DEBUG: Task 1051 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1052 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1052 submitted for App calibrate, waiting on task 1051
parsl.dataflow.dflow DEBUG: Task 1052 set to pending state with AppFuture: <AppFuture at 0x14aa155de410 state=pending>
parsl.dataflow.dflow DEBUG: Task 1052 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1053 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1053 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1053 set to pending state with AppFuture: <AppFuture at 0x14aa155df0d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1053 has memoization hash e0c477f5b4c829b878ef86c7d43d71bd
parsl.dataflow.memoization INFO: Task 1053 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1053 try 0 launched on executor multi with executor id 352
parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/5b528d9b-6e29-46ee-b653-d19982f317f7_isr_855821_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/5b528d9b-6e29-46ee-b653-d19982f317f7_isr_855821_1.stderr
parsl.dataflow.dflow DEBUG: Task 1054 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1054 submitted for App characterizeImage, waiting on task 1053
parsl.dataflow.dflow DEBUG: Task 1054 set to pending state with AppFuture: <AppFuture at 0x14aa155df450 state=pending>
parsl.dataflow.dflow DEBUG: Task 1054 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1055 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1055 submitted for App calibrate, waiting on task 1054
parsl.dataflow.dflow DEBUG: Task 1055 set to pending state with AppFuture: <AppFuture at 0x14aa155dfcd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1055 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1056 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1056 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1056 set to pending state with AppFuture: <AppFuture at 0x14aa155e85d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1056 has memoization hash c2ea701eeaff248535aad108c508d4f8
parsl.dataflow.memoization INFO: Task 1056 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1056 try 0 launched on executor multi with executor id 353
parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/eb668132-1857-4ac5-906f-5f924c1d511d_isr_855910_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/eb668132-1857-4ac5-906f-5f924c1d511d_isr_855910_48.stderr
parsl.dataflow.dflow DEBUG: Task 1057 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1057 submitted for App characterizeImage, waiting on task 1056
parsl.dataflow.dflow DEBUG: Task 1057 set to pending state with AppFuture: <AppFuture at 0x14aa155de7d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1057 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1058 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1058 submitted for App calibrate, waiting on task 1057
parsl.dataflow.dflow DEBUG: Task 1058 set to pending state with AppFuture: <AppFuture at 0x14aa155e9490 state=pending>
parsl.dataflow.dflow DEBUG: Task 1058 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1059 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1059 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1059 set to pending state with AppFuture: <AppFuture at 0x14aa155e9990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1059 has memoization hash 64f0d1156183836ff98e741689908c75
parsl.dataflow.memoization INFO: Task 1059 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1059 try 0 launched on executor multi with executor id 354
parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/a4796649-3a06-4cba-a7b9-6f2fe1810990_isr_855822_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/a4796649-3a06-4cba-a7b9-6f2fe1810990_isr_855822_38.stderr
parsl.dataflow.dflow DEBUG: Task 1060 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1060 submitted for App characterizeImage, waiting on task 1059
parsl.dataflow.dflow DEBUG: Task 1060 set to pending state with AppFuture: <AppFuture at 0x14aa155ea950 state=pending>
parsl.dataflow.dflow DEBUG: Task 1060 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1061 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1061 submitted for App calibrate, waiting on task 1060
parsl.dataflow.dflow DEBUG: Task 1061 set to pending state with AppFuture: <AppFuture at 0x14aa155ea650 state=pending>
parsl.dataflow.dflow DEBUG: Task 1061 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1062 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1062 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1062 set to pending state with AppFuture: <AppFuture at 0x14aa155eae50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1062 has memoization hash 031d0f751d155af1b41e4a26e21eb252
parsl.dataflow.memoization INFO: Task 1062 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1062 try 0 launched on executor multi with executor id 355
parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/6995e03f-f5cc-49e0-815e-ae3d7e093079_isr_855821_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/6995e03f-f5cc-49e0-815e-ae3d7e093079_isr_855821_17.stderr
parsl.dataflow.dflow DEBUG: Task 1063 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1063 submitted for App characterizeImage, waiting on task 1062
parsl.dataflow.dflow DEBUG: Task 1063 set to pending state with AppFuture: <AppFuture at 0x14aa155eb490 state=pending>
parsl.dataflow.dflow DEBUG: Task 1063 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1064 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1064 submitted for App calibrate, waiting on task 1063
parsl.dataflow.dflow DEBUG: Task 1064 set to pending state with AppFuture: <AppFuture at 0x14aa155ebe90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1064 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1065 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1065 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1065 set to pending state with AppFuture: <AppFuture at 0x14aa155f0310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1065 has memoization hash ba30bc03a8bff5e4a957a8217ad4f41d
parsl.dataflow.memoization INFO: Task 1065 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1065 try 0 launched on executor multi with executor id 356
parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/cf761848-8b9a-4732-a8d9-9419a6e935cf_isr_855910_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/cf761848-8b9a-4732-a8d9-9419a6e935cf_isr_855910_31.stderr
parsl.dataflow.dflow DEBUG: Task 1066 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1066 submitted for App characterizeImage, waiting on task 1065
parsl.dataflow.dflow DEBUG: Task 1066 set to pending state with AppFuture: <AppFuture at 0x14aa157a1190 state=pending>
parsl.dataflow.dflow DEBUG: Task 1066 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1067 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1067 submitted for App calibrate, waiting on task 1066
parsl.dataflow.dflow DEBUG: Task 1067 set to pending state with AppFuture: <AppFuture at 0x14aa155f1150 state=pending>
parsl.dataflow.dflow DEBUG: Task 1067 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1068 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1068 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1068 set to pending state with AppFuture: <AppFuture at 0x14aa155f1710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1068 has memoization hash 5497fd364b646ad094501e22de137513
parsl.dataflow.memoization INFO: Task 1068 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1068 try 0 launched on executor multi with executor id 357
parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/ac673ddf-c87f-4bd0-84be-e0163097f0ec_isr_855821_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/ac673ddf-c87f-4bd0-84be-e0163097f0ec_isr_855821_52.stderr
parsl.dataflow.dflow DEBUG: Task 1069 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1069 submitted for App characterizeImage, waiting on task 1068
parsl.dataflow.dflow DEBUG: Task 1069 set to pending state with AppFuture: <AppFuture at 0x14aa155f26d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1069 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1070 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1070 submitted for App calibrate, waiting on task 1069
parsl.dataflow.dflow DEBUG: Task 1070 set to pending state with AppFuture: <AppFuture at 0x14aa155f1bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1070 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1071 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1071 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1071 set to pending state with AppFuture: <AppFuture at 0x14aa155f2ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1071 has memoization hash 894996dd2dee8346d1113021a37e4ff4
parsl.dataflow.memoization INFO: Task 1071 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1071 try 0 launched on executor multi with executor id 358
parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/2ede4735-0950-497c-a70b-5316d01523bd_isr_855910_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/2ede4735-0950-497c-a70b-5316d01523bd_isr_855910_26.stderr
parsl.dataflow.dflow DEBUG: Task 1072 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1072 submitted for App characterizeImage, waiting on task 1071
parsl.dataflow.dflow DEBUG: Task 1072 set to pending state with AppFuture: <AppFuture at 0x14aa155f3250 state=pending>
parsl.dataflow.dflow DEBUG: Task 1072 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1073 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1073 submitted for App calibrate, waiting on task 1072
parsl.dataflow.dflow DEBUG: Task 1073 set to pending state with AppFuture: <AppFuture at 0x14aa157a2510 state=pending>
parsl.dataflow.dflow DEBUG: Task 1073 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1074 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1074 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1074 set to pending state with AppFuture: <AppFuture at 0x14aa155f3e90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1074 has memoization hash 8bf80a11f72601965c8a22c36b7809e1
parsl.dataflow.memoization INFO: Task 1074 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1074 try 0 launched on executor multi with executor id 359
parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/13d0bbad-12ca-4696-a159-500990c0dafe_isr_855820_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/13d0bbad-12ca-4696-a159-500990c0dafe_isr_855820_12.stderr
parsl.dataflow.dflow DEBUG: Task 1075 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1075 submitted for App characterizeImage, waiting on task 1074
parsl.dataflow.dflow DEBUG: Task 1075 set to pending state with AppFuture: <AppFuture at 0x14aa155f31d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1075 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1076 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1076 submitted for App calibrate, waiting on task 1075
parsl.dataflow.dflow DEBUG: Task 1076 set to pending state with AppFuture: <AppFuture at 0x14aa155fc6d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1076 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1077 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1077 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1077 set to pending state with AppFuture: <AppFuture at 0x14aa155fd250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1077 has memoization hash 466dedf4d6215d543575cef014fc39ca
parsl.dataflow.memoization INFO: Task 1077 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1077 try 0 launched on executor multi with executor id 360
parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/252286b1-b28f-4857-be29-06bc8a0f74fa_isr_855911_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/252286b1-b28f-4857-be29-06bc8a0f74fa_isr_855911_9.stderr
parsl.dataflow.dflow DEBUG: Task 1078 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1078 submitted for App characterizeImage, waiting on task 1077
parsl.dataflow.dflow DEBUG: Task 1078 set to pending state with AppFuture: <AppFuture at 0x14aa155fd3d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1078 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1079 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1079 submitted for App calibrate, waiting on task 1078
parsl.dataflow.dflow DEBUG: Task 1079 set to pending state with AppFuture: <AppFuture at 0x14aa155fe190 state=pending>
parsl.dataflow.dflow DEBUG: Task 1079 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1080 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1080 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1080 set to pending state with AppFuture: <AppFuture at 0x14aa155fe690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1080 has memoization hash a7394b50ccb9928e9ee85986a626ae28
parsl.dataflow.memoization INFO: Task 1080 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1080 try 0 launched on executor multi with executor id 361
parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/9d79b5c0-1302-4222-84aa-294315783e8c_isr_855820_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/9d79b5c0-1302-4222-84aa-294315783e8c_isr_855820_49.stderr
parsl.dataflow.dflow DEBUG: Task 1081 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1081 submitted for App characterizeImage, waiting on task 1080
parsl.dataflow.dflow DEBUG: Task 1081 set to pending state with AppFuture: <AppFuture at 0x14aa155fda50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1081 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1082 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1082 submitted for App calibrate, waiting on task 1081
parsl.dataflow.dflow DEBUG: Task 1082 set to pending state with AppFuture: <AppFuture at 0x14aa155ff110 state=pending>
parsl.dataflow.dflow DEBUG: Task 1082 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1083 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1083 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1083 set to pending state with AppFuture: <AppFuture at 0x14aa155ffa90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1083 has memoization hash 265802a4864d78d954b5928a3ec9b704
parsl.dataflow.memoization INFO: Task 1083 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1083 try 0 launched on executor multi with executor id 362
parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/2386f827-d9d7-43c3-9201-36565f5391a2_isr_855822_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/2386f827-d9d7-43c3-9201-36565f5391a2_isr_855822_28.stderr
parsl.dataflow.dflow DEBUG: Task 1084 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1084 submitted for App characterizeImage, waiting on task 1083
parsl.dataflow.dflow DEBUG: Task 1084 set to pending state with AppFuture: <AppFuture at 0x14aa155ffd90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1084 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1085 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1085 submitted for App calibrate, waiting on task 1084
parsl.dataflow.dflow DEBUG: Task 1085 set to pending state with AppFuture: <AppFuture at 0x14aa15604950 state=pending>
parsl.dataflow.dflow DEBUG: Task 1085 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1086 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1086 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1086 set to pending state with AppFuture: <AppFuture at 0x14aa15604ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1086 has memoization hash 28fa4c73fec77fcf8669195cac2d3f5b
parsl.dataflow.memoization INFO: Task 1086 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1086 try 0 launched on executor multi with executor id 363
parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/3ecd8a7d-a861-4335-9c17-6a784c363614_isr_855821_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/3ecd8a7d-a861-4335-9c17-6a784c363614_isr_855821_62.stderr
parsl.dataflow.dflow DEBUG: Task 1087 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1087 submitted for App characterizeImage, waiting on task 1086
parsl.dataflow.dflow DEBUG: Task 1087 set to pending state with AppFuture: <AppFuture at 0x14aa15604c10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1087 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1088 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1088 submitted for App calibrate, waiting on task 1087
parsl.dataflow.dflow DEBUG: Task 1088 set to pending state with AppFuture: <AppFuture at 0x14aa15606010 state=pending>
parsl.dataflow.dflow DEBUG: Task 1088 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1089 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1089 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1089 set to pending state with AppFuture: <AppFuture at 0x14aa15606290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1089 has memoization hash 2019765511fbd0a99dd475d4c473969f
parsl.dataflow.memoization INFO: Task 1089 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1089 try 0 launched on executor multi with executor id 364
parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/768a8a59-1444-4964-b81f-712074234c8e_isr_855820_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/768a8a59-1444-4964-b81f-712074234c8e_isr_855820_14.stderr
parsl.dataflow.dflow DEBUG: Task 1090 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1090 submitted for App characterizeImage, waiting on task 1089
parsl.dataflow.dflow DEBUG: Task 1090 set to pending state with AppFuture: <AppFuture at 0x14aa15605990 state=pending>
parsl.dataflow.dflow DEBUG: Task 1090 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1091 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1091 submitted for App calibrate, waiting on task 1090
parsl.dataflow.dflow DEBUG: Task 1091 set to pending state with AppFuture: <AppFuture at 0x14aa15606b10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1091 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1092 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1092 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1092 set to pending state with AppFuture: <AppFuture at 0x14aa15607650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1092 has memoization hash 6751151623c655e1ae7e751db1d8f2d5
parsl.dataflow.memoization INFO: Task 1092 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1092 try 0 launched on executor multi with executor id 365
parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/01404079-48a7-4757-b2d7-dc5816702d15_isr_855820_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/01404079-48a7-4757-b2d7-dc5816702d15_isr_855820_4.stderr
parsl.dataflow.dflow DEBUG: Task 1093 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1093 submitted for App characterizeImage, waiting on task 1092
parsl.dataflow.dflow DEBUG: Task 1093 set to pending state with AppFuture: <AppFuture at 0x14aa15604510 state=pending>
parsl.dataflow.dflow DEBUG: Task 1093 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1094 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1094 submitted for App calibrate, waiting on task 1093
parsl.dataflow.dflow DEBUG: Task 1094 set to pending state with AppFuture: <AppFuture at 0x14aa15610550 state=pending>
parsl.dataflow.dflow DEBUG: Task 1094 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1095 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1095 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1095 set to pending state with AppFuture: <AppFuture at 0x14aa15610a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1095 has memoization hash f58c17aa9ddd7e9b57eeea70c88d6e83
parsl.dataflow.memoization INFO: Task 1095 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1560c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1095 try 0 launched on executor multi with executor id 366
parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/23e7ec59-1506-49eb-82bf-a63464c2a073_isr_855820_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/23e7ec59-1506-49eb-82bf-a63464c2a073_isr_855820_22.stderr
parsl.dataflow.dflow DEBUG: Task 1096 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1096 submitted for App characterizeImage, waiting on task 1095
parsl.dataflow.dflow DEBUG: Task 1096 set to pending state with AppFuture: <AppFuture at 0x14aa158912d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1096 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1097 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1097 submitted for App calibrate, waiting on task 1096
parsl.dataflow.dflow DEBUG: Task 1097 set to pending state with AppFuture: <AppFuture at 0x14aa15611950 state=pending>
parsl.dataflow.dflow DEBUG: Task 1097 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1098 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1098 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1098 set to pending state with AppFuture: <AppFuture at 0x14aa15611e90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1098 has memoization hash 0672afa3a2fb647706aee055770f87c4
parsl.dataflow.memoization INFO: Task 1098 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1560c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1098 try 0 launched on executor multi with executor id 367
parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/ede0b153-4c6b-457f-9518-d480953443a9_isr_855910_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/ede0b153-4c6b-457f-9518-d480953443a9_isr_855910_18.stderr
parsl.dataflow.dflow DEBUG: Task 1099 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1099 submitted for App characterizeImage, waiting on task 1098
parsl.dataflow.dflow DEBUG: Task 1099 set to pending state with AppFuture: <AppFuture at 0x14aa15611210 state=pending>
parsl.dataflow.dflow DEBUG: Task 1099 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1100 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1100 submitted for App calibrate, waiting on task 1099
parsl.dataflow.dflow DEBUG: Task 1100 set to pending state with AppFuture: <AppFuture at 0x14aa15612dd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1100 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1101 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1101 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1101 set to pending state with AppFuture: <AppFuture at 0x14aa15613310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1101 has memoization hash 69735f6ea01adcc496ecec1cfa6653ea
parsl.dataflow.memoization INFO: Task 1101 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1560c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1101 try 0 launched on executor multi with executor id 368
parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/2ab875e5-4a08-4db0-861e-4e39f1c39a58_isr_855820_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/2ab875e5-4a08-4db0-861e-4e39f1c39a58_isr_855820_51.stderr
parsl.dataflow.dflow DEBUG: Task 1102 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1102 submitted for App characterizeImage, waiting on task 1101
parsl.dataflow.dflow DEBUG: Task 1102 set to pending state with AppFuture: <AppFuture at 0x14aa156139d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1102 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1103 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1103 submitted for App calibrate, waiting on task 1102
parsl.dataflow.dflow DEBUG: Task 1103 set to pending state with AppFuture: <AppFuture at 0x14aa157c1450 state=pending>
parsl.dataflow.dflow DEBUG: Task 1103 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1104 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1104 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1104 set to pending state with AppFuture: <AppFuture at 0x14aa15624790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1104 has memoization hash e714b37d17002eebe1db94bd02d860df
parsl.dataflow.memoization INFO: Task 1104 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1560cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1104 try 0 launched on executor multi with executor id 369
parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/b7beb70e-baa8-4ce1-a755-a20d4988b0ad_isr_855910_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/b7beb70e-baa8-4ce1-a755-a20d4988b0ad_isr_855910_58.stderr
parsl.dataflow.dflow DEBUG: Task 1105 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1105 submitted for App characterizeImage, waiting on task 1104
parsl.dataflow.dflow DEBUG: Task 1105 set to pending state with AppFuture: <AppFuture at 0x14aa15624910 state=pending>
parsl.dataflow.dflow DEBUG: Task 1105 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1106 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1106 submitted for App calibrate, waiting on task 1105
parsl.dataflow.dflow DEBUG: Task 1106 set to pending state with AppFuture: <AppFuture at 0x14aa15625750 state=pending>
parsl.dataflow.dflow DEBUG: Task 1106 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1107 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1107 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1107 set to pending state with AppFuture: <AppFuture at 0x14aa15625b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1107 has memoization hash c6489b52da513eac6575a7e8d35c5474
parsl.dataflow.memoization INFO: Task 1107 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1560cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1107 try 0 launched on executor multi with executor id 370
parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/1e844243-0f43-41df-85ca-ab0b54c618e8_isr_855911_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/1e844243-0f43-41df-85ca-ab0b54c618e8_isr_855911_56.stderr
parsl.dataflow.dflow DEBUG: Task 1108 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1108 submitted for App characterizeImage, waiting on task 1107
parsl.dataflow.dflow DEBUG: Task 1108 set to pending state with AppFuture: <AppFuture at 0x14aa15625cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1108 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1109 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1109 submitted for App calibrate, waiting on task 1108
parsl.dataflow.dflow DEBUG: Task 1109 set to pending state with AppFuture: <AppFuture at 0x14aa15626590 state=pending>
parsl.dataflow.dflow DEBUG: Task 1109 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1110 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1110 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1110 set to pending state with AppFuture: <AppFuture at 0x14aa15626f10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1110 has memoization hash 4f1c424b4bac8838728fe2d029a887e3
parsl.dataflow.memoization INFO: Task 1110 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1560d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1110 try 0 launched on executor multi with executor id 371
parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/b22dae0c-c28c-43e3-8921-9aa1bc8fd8ab_isr_855910_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/b22dae0c-c28c-43e3-8921-9aa1bc8fd8ab_isr_855910_24.stderr
parsl.dataflow.dflow DEBUG: Task 1111 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1111 submitted for App characterizeImage, waiting on task 1110
parsl.dataflow.dflow DEBUG: Task 1111 set to pending state with AppFuture: <AppFuture at 0x14aa15626c50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1111 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1112 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1112 submitted for App calibrate, waiting on task 1111
parsl.dataflow.dflow DEBUG: Task 1112 set to pending state with AppFuture: <AppFuture at 0x14aa15627e10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1112 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1113 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1113 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1113 set to pending state with AppFuture: <AppFuture at 0x14aa1562c390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1113 has memoization hash cc70999d306a4d9349f74e3cba9dfd94
parsl.dataflow.memoization INFO: Task 1113 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1560d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1113 try 0 launched on executor multi with executor id 372
parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/05d4f8da-75dd-49e6-b3a9-e0337c23d0c1_isr_855822_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/05d4f8da-75dd-49e6-b3a9-e0337c23d0c1_isr_855822_14.stderr
parsl.dataflow.dflow DEBUG: Task 1114 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1114 submitted for App characterizeImage, waiting on task 1113
parsl.dataflow.dflow DEBUG: Task 1114 set to pending state with AppFuture: <AppFuture at 0x14aa156ecbd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1114 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1115 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1115 submitted for App calibrate, waiting on task 1114
parsl.dataflow.dflow DEBUG: Task 1115 set to pending state with AppFuture: <AppFuture at 0x14aa1562cc50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1115 has outstanding dependencies, so launch_if_ready skipping
2024-03-09 20:54:31 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 372
parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism
parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out
parsl.dataflow.strategy DEBUG: Requesting 92 more blocks
parsl.executors.status_handling INFO: Scaling out by 92 blocks
parsl.executors.status_handling INFO: Allocated block ID 1
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1
2024-03-09 20:54:31 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-1
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-1
2024-03-09 20:54:31 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:31 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:31 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:31 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:31 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-09 20:54:31 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:31 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:32 proc_lsst.multi:170 [INFO]  [multi] job_id 16968884
proc_lsst.multi INFO: [multi] job_id 16968884
2024-03-09 20:54:32 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:32 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:32 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968884
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968884
parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 16968884
parsl.executors.status_handling INFO: Allocated block ID 2
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2
2024-03-09 20:54:32 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-2
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-2
2024-03-09 20:54:32 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:32 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:32 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:32 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:32 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:32 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:32 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:32 proc_lsst.multi:170 [INFO]  [multi] job_id 16968885
proc_lsst.multi INFO: [multi] job_id 16968885
2024-03-09 20:54:32 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-09 20:54:32 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:32 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968885
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968885
parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 16968885
parsl.executors.status_handling INFO: Allocated block ID 3
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3
2024-03-09 20:54:32 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-3
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-3
2024-03-09 20:54:32 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:32 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:32 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:32 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:32 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-09 20:54:32 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:32 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:33 proc_lsst.multi:170 [INFO]  [multi] job_id 16968886
proc_lsst.multi INFO: [multi] job_id 16968886
2024-03-09 20:54:33 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-09 20:54:33 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:33 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968886
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968886
parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 16968886
parsl.executors.status_handling INFO: Allocated block ID 4
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4
2024-03-09 20:54:33 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-4
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-4
2024-03-09 20:54:33 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:33 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-09 20:54:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:33 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:33 proc_lsst.multi:170 [INFO]  [multi] job_id 16968887
proc_lsst.multi INFO: [multi] job_id 16968887
2024-03-09 20:54:33 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-09 20:54:33 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:33 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968887
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968887
parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 16968887
parsl.executors.status_handling INFO: Allocated block ID 5
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5
2024-03-09 20:54:33 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-5
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-5
2024-03-09 20:54:33 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:33 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-09 20:54:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:33 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:33 proc_lsst.multi:170 [INFO]  [multi] job_id 16968888
proc_lsst.multi INFO: [multi] job_id 16968888
2024-03-09 20:54:33 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:33 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:33 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968888
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968888
parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 16968888
parsl.executors.status_handling INFO: Allocated block ID 6
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6
2024-03-09 20:54:33 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-6
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-6
2024-03-09 20:54:33 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:33 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:33 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:34 proc_lsst.multi:170 [INFO]  [multi] job_id 16968889
proc_lsst.multi INFO: [multi] job_id 16968889
2024-03-09 20:54:34 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-09 20:54:34 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:34 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968889
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968889
parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 16968889
parsl.executors.status_handling INFO: Allocated block ID 7
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7
2024-03-09 20:54:34 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-7
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-7
2024-03-09 20:54:34 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:34 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-09 20:54:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:34 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:34 proc_lsst.multi:170 [INFO]  [multi] job_id 16968890
proc_lsst.multi INFO: [multi] job_id 16968890
2024-03-09 20:54:34 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-09 20:54:34 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:34 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968890
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968890
parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 16968890
parsl.executors.status_handling INFO: Allocated block ID 8
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8
2024-03-09 20:54:34 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-8
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-8
2024-03-09 20:54:34 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:34 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-09 20:54:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:34 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:35 proc_lsst.multi:170 [INFO]  [multi] job_id 16968891
proc_lsst.multi INFO: [multi] job_id 16968891
2024-03-09 20:54:35 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-09 20:54:35 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:35 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968891
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968891
parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 16968891
parsl.executors.status_handling INFO: Allocated block ID 9
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9
2024-03-09 20:54:35 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-9
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-9
2024-03-09 20:54:35 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:35 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-09 20:54:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:35 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:35 proc_lsst.multi:170 [INFO]  [multi] job_id 16968892
proc_lsst.multi INFO: [multi] job_id 16968892
2024-03-09 20:54:35 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-09 20:54:35 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:35 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968892
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968892
parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 16968892
parsl.executors.status_handling INFO: Allocated block ID 10
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10
2024-03-09 20:54:35 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-10
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-10
2024-03-09 20:54:35 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:35 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-09 20:54:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:35 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:36 proc_lsst.multi:170 [INFO]  [multi] job_id 16968893
proc_lsst.multi INFO: [multi] job_id 16968893
2024-03-09 20:54:36 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-09 20:54:36 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:36 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968893
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968893
parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 16968893
parsl.executors.status_handling INFO: Allocated block ID 11
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11
2024-03-09 20:54:36 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-11
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-11
2024-03-09 20:54:36 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:36 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-09 20:54:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:36 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:36 proc_lsst.multi:170 [INFO]  [multi] job_id 16968894
proc_lsst.multi INFO: [multi] job_id 16968894
2024-03-09 20:54:36 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-09 20:54:36 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:36 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968894
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968894
parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 16968894
parsl.executors.status_handling INFO: Allocated block ID 12
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12
2024-03-09 20:54:36 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-12
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-12
2024-03-09 20:54:36 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:36 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-09 20:54:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:36 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:37 proc_lsst.multi:170 [INFO]  [multi] job_id 16968895
proc_lsst.multi INFO: [multi] job_id 16968895
2024-03-09 20:54:37 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-09 20:54:37 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:37 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968895
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968895
parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 16968895
parsl.executors.status_handling INFO: Allocated block ID 13
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13
2024-03-09 20:54:37 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-13
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-13
2024-03-09 20:54:37 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:37 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-09 20:54:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:37 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:38 proc_lsst.multi:170 [INFO]  [multi] job_id 16968896
proc_lsst.multi INFO: [multi] job_id 16968896
2024-03-09 20:54:38 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-09 20:54:38 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:38 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968896
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968896
parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 16968896
parsl.executors.status_handling INFO: Allocated block ID 14
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14
2024-03-09 20:54:38 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-14
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-14
2024-03-09 20:54:38 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:38 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-09 20:54:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:38 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:38 proc_lsst.multi:170 [INFO]  [multi] job_id 16968897
proc_lsst.multi INFO: [multi] job_id 16968897
2024-03-09 20:54:38 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-09 20:54:38 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:38 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968897
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968897
parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 16968897
parsl.executors.status_handling INFO: Allocated block ID 15
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15
2024-03-09 20:54:38 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-15
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-15
2024-03-09 20:54:38 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:38 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-09 20:54:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:38 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:39 proc_lsst.multi:170 [INFO]  [multi] job_id 16968898
proc_lsst.multi INFO: [multi] job_id 16968898
2024-03-09 20:54:39 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-09 20:54:39 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:39 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968898
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968898
parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 16968898
parsl.executors.status_handling INFO: Allocated block ID 16
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16
2024-03-09 20:54:39 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-16
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-16
2024-03-09 20:54:39 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:39 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-09 20:54:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:39 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:39 proc_lsst.multi:170 [INFO]  [multi] job_id 16968899
proc_lsst.multi INFO: [multi] job_id 16968899
2024-03-09 20:54:39 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-09 20:54:39 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:39 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968899
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968899
parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 16968899
parsl.executors.status_handling INFO: Allocated block ID 17
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17
2024-03-09 20:54:39 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-17
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-17
2024-03-09 20:54:39 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:39 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-09 20:54:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:39 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:39 proc_lsst.multi:170 [INFO]  [multi] job_id 16968900
proc_lsst.multi INFO: [multi] job_id 16968900
2024-03-09 20:54:39 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-09 20:54:39 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:39 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968900
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968900
parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 16968900
parsl.executors.status_handling INFO: Allocated block ID 18
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18
2024-03-09 20:54:39 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-18
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-18
2024-03-09 20:54:39 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:39 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-09 20:54:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:39 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:40 proc_lsst.multi:170 [INFO]  [multi] job_id 16968901
proc_lsst.multi INFO: [multi] job_id 16968901
2024-03-09 20:54:40 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-09 20:54:40 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:40 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968901
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968901
parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 16968901
parsl.executors.status_handling INFO: Allocated block ID 19
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19
2024-03-09 20:54:40 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-19
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-19
2024-03-09 20:54:40 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:40 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-09 20:54:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:40 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:40 proc_lsst.multi:170 [INFO]  [multi] job_id 16968902
proc_lsst.multi INFO: [multi] job_id 16968902
2024-03-09 20:54:40 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-09 20:54:40 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:40 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968902
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968902
parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 16968902
parsl.executors.status_handling INFO: Allocated block ID 20
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20
2024-03-09 20:54:40 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-20
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-20
2024-03-09 20:54:40 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:40 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-09 20:54:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:40 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:40 proc_lsst.multi:170 [INFO]  [multi] job_id 16968903
proc_lsst.multi INFO: [multi] job_id 16968903
2024-03-09 20:54:40 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:54:40 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:40 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968903
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968903
parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 16968903
parsl.executors.status_handling INFO: Allocated block ID 21
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21
2024-03-09 20:54:40 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-21
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-21
2024-03-09 20:54:40 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:40 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:54:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:40 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:41 proc_lsst.multi:170 [INFO]  [multi] job_id 16968904
proc_lsst.multi INFO: [multi] job_id 16968904
2024-03-09 20:54:41 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-09 20:54:41 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:41 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968904
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968904
parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 16968904
parsl.executors.status_handling INFO: Allocated block ID 22
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22
2024-03-09 20:54:41 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-22
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-22
2024-03-09 20:54:41 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:41 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-09 20:54:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:41 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:41 proc_lsst.multi:170 [INFO]  [multi] job_id 16968905
proc_lsst.multi INFO: [multi] job_id 16968905
2024-03-09 20:54:41 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-09 20:54:41 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:41 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968905
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968905
parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 16968905
parsl.executors.status_handling INFO: Allocated block ID 23
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23
2024-03-09 20:54:41 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-23
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-23
2024-03-09 20:54:41 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:41 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-09 20:54:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:41 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:41 proc_lsst.multi:170 [INFO]  [multi] job_id 16968906
proc_lsst.multi INFO: [multi] job_id 16968906
2024-03-09 20:54:41 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-09 20:54:41 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:41 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968906
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968906
parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 16968906
parsl.executors.status_handling INFO: Allocated block ID 24
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24
2024-03-09 20:54:41 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-24
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-24
2024-03-09 20:54:41 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:41 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-09 20:54:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:41 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:42 proc_lsst.multi:170 [INFO]  [multi] job_id 16968907
proc_lsst.multi INFO: [multi] job_id 16968907
2024-03-09 20:54:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-09 20:54:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:42 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968907
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968907
parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 16968907
parsl.executors.status_handling INFO: Allocated block ID 25
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25
2024-03-09 20:54:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-25
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-25
2024-03-09 20:54:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-09 20:54:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:42 proc_lsst.multi:170 [INFO]  [multi] job_id 16968909
proc_lsst.multi INFO: [multi] job_id 16968909
2024-03-09 20:54:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-09 20:54:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:42 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968909
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968909
parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 16968909
parsl.executors.status_handling INFO: Allocated block ID 26
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26
2024-03-09 20:54:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-26
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-26
2024-03-09 20:54:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-09 20:54:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:43 proc_lsst.multi:170 [INFO]  [multi] job_id 16968911
proc_lsst.multi INFO: [multi] job_id 16968911
2024-03-09 20:54:43 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-09 20:54:43 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:43 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968911
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968911
parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 16968911
parsl.executors.status_handling INFO: Allocated block ID 27
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27
2024-03-09 20:54:43 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-27
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-27
2024-03-09 20:54:43 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:43 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-09 20:54:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:43 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:43 proc_lsst.multi:170 [INFO]  [multi] job_id 16968912
proc_lsst.multi INFO: [multi] job_id 16968912
2024-03-09 20:54:43 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-09 20:54:43 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:43 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968912
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968912
parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 16968912
parsl.executors.status_handling INFO: Allocated block ID 28
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28
2024-03-09 20:54:43 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-28
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-28
2024-03-09 20:54:43 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:43 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-09 20:54:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:43 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:43 proc_lsst.multi:170 [INFO]  [multi] job_id 16968913
proc_lsst.multi INFO: [multi] job_id 16968913
2024-03-09 20:54:43 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-09 20:54:43 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:43 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968913
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968913
parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 16968913
parsl.executors.status_handling INFO: Allocated block ID 29
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29
2024-03-09 20:54:43 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-29
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-29
2024-03-09 20:54:43 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:43 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-09 20:54:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:43 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:46 proc_lsst.multi:170 [INFO]  [multi] job_id 16968917
proc_lsst.multi INFO: [multi] job_id 16968917
2024-03-09 20:54:46 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-09 20:54:46 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:46 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968917
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968917
parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 16968917
parsl.executors.status_handling INFO: Allocated block ID 30
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30
2024-03-09 20:54:46 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-30
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-30
2024-03-09 20:54:46 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:46 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-09 20:54:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:46 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:46 proc_lsst.multi:170 [INFO]  [multi] job_id 16968918
proc_lsst.multi INFO: [multi] job_id 16968918
2024-03-09 20:54:46 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:46 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:46 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16968918
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968918
parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 16968918
parsl.executors.status_handling INFO: Allocated block ID 31
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31
2024-03-09 20:54:46 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-31
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-31
2024-03-09 20:54:46 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:46 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:46 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-09 20:54:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:46 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:47 proc_lsst.multi:170 [INFO]  [multi] job_id 16968919
proc_lsst.multi INFO: [multi] job_id 16968919
2024-03-09 20:54:47 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:47 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:47 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 16968919
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16968919
parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 16968919
parsl.executors.status_handling INFO: Allocated block ID 32
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32
2024-03-09 20:54:47 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-32
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-32
2024-03-09 20:54:47 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:47 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:47 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:47 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:47 proc_lsst.multi:170 [INFO]  [multi] job_id 16968920
proc_lsst.multi INFO: [multi] job_id 16968920
2024-03-09 20:54:47 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-09 20:54:47 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:47 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 16968920
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16968920
parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 16968920
parsl.executors.status_handling INFO: Allocated block ID 33
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33
2024-03-09 20:54:47 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-33
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-33
2024-03-09 20:54:47 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:47 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:47 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-09 20:54:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:47 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:48 proc_lsst.multi:170 [INFO]  [multi] job_id 16968921
proc_lsst.multi INFO: [multi] job_id 16968921
2024-03-09 20:54:48 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-09 20:54:48 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:48 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 16968921
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16968921
parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 16968921
parsl.executors.status_handling INFO: Allocated block ID 34
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34
2024-03-09 20:54:48 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-34
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-34
2024-03-09 20:54:48 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:48 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:48 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-09 20:54:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:48 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:48 proc_lsst.multi:170 [INFO]  [multi] job_id 16968922
proc_lsst.multi INFO: [multi] job_id 16968922
2024-03-09 20:54:48 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-09 20:54:48 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:48 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 16968922
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16968922
parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 16968922
parsl.executors.status_handling INFO: Allocated block ID 35
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35
2024-03-09 20:54:48 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-35
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-35
2024-03-09 20:54:48 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:48 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:48 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-09 20:54:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:48 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:49 proc_lsst.multi:170 [INFO]  [multi] job_id 16968923
proc_lsst.multi INFO: [multi] job_id 16968923
2024-03-09 20:54:49 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:49 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:49 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 16968923
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16968923
parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 16968923
parsl.executors.status_handling INFO: Allocated block ID 36
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36
2024-03-09 20:54:49 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-36
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-36
2024-03-09 20:54:49 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:49 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:49 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:49 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-09 20:54:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:49 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9c7cd10186f7f718f805aa968a232687 with result from task 3
parsl.dataflow.dflow DEBUG: Task 3 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 9eec1aeaf65e20888775993fac73ff1f
parsl.dataflow.memoization INFO: Task 4 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fcb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 373
parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/3876bf01-95f0-4738-8e5d-a3052cf869b8_characterizeImage_855911_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/3876bf01-95f0-4738-8e5d-a3052cf869b8_characterizeImage_855911_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/2ca5dd9a-a895-47e7-b6af-063c90df40d0_isr_855911_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/2ca5dd9a-a895-47e7-b6af-063c90df40d0_isr_855911_33.stderr
2024-03-09 20:54:49 proc_lsst.multi:170 [INFO]  [multi] job_id 16968924
proc_lsst.multi INFO: [multi] job_id 16968924
2024-03-09 20:54:49 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:49 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:49 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16968924
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968924
parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 16968924
parsl.executors.status_handling INFO: Allocated block ID 37
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37
2024-03-09 20:54:49 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-37
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-37
2024-03-09 20:54:49 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:49 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:49 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:49 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:49 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:50 proc_lsst.multi:170 [INFO]  [multi] job_id 16968925
proc_lsst.multi INFO: [multi] job_id 16968925
2024-03-09 20:54:50 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-09 20:54:50 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:50 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16968925
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968925
parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 16968925
parsl.executors.status_handling INFO: Allocated block ID 38
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38
2024-03-09 20:54:50 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-38
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-38
2024-03-09 20:54:50 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:50 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:50 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:50 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-09 20:54:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:50 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:51 proc_lsst.multi:170 [INFO]  [multi] job_id 16968926
proc_lsst.multi INFO: [multi] job_id 16968926
2024-03-09 20:54:51 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-09 20:54:51 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:51 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16968926
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968926
parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 16968926
parsl.executors.status_handling INFO: Allocated block ID 39
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39
2024-03-09 20:54:51 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-39
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-39
2024-03-09 20:54:51 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:51 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:51 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:51 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-09 20:54:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:51 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c71f2ec4bc90d67631ec679fc947ae05 with result from task 0
parsl.dataflow.dflow DEBUG: Task 0 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1 has memoization hash ab81aeee09894bbd19504a32c3fd4331
parsl.dataflow.memoization INFO: Task 1 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fc4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 374
parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/d5f119b6-af11-4fa0-a7dd-94e6ecfeb938_characterizeImage_855822_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/d5f119b6-af11-4fa0-a7dd-94e6ecfeb938_characterizeImage_855822_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/bc39d58e-287f-4f59-8466-6cb8ccc18d09_isr_855822_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/bc39d58e-287f-4f59-8466-6cb8ccc18d09_isr_855822_22.stderr
2024-03-09 20:54:51 proc_lsst.multi:170 [INFO]  [multi] job_id 16968927
proc_lsst.multi INFO: [multi] job_id 16968927
2024-03-09 20:54:51 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-09 20:54:51 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:51 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16968927
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968927
parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 16968927
parsl.executors.status_handling INFO: Allocated block ID 40
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40
2024-03-09 20:54:51 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-40
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-40
2024-03-09 20:54:51 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:51 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:51 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:51 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-09 20:54:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:51 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:52 proc_lsst.multi:170 [INFO]  [multi] job_id 16968928
proc_lsst.multi INFO: [multi] job_id 16968928
2024-03-09 20:54:52 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:52 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:52 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16968928
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968928
parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 16968928
parsl.executors.status_handling INFO: Allocated block ID 41
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41
2024-03-09 20:54:52 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-41
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-41
2024-03-09 20:54:52 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:52 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:52 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:52 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:52 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:52 proc_lsst.multi:170 [INFO]  [multi] job_id 16968929
proc_lsst.multi INFO: [multi] job_id 16968929
2024-03-09 20:54:52 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-09 20:54:52 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:52 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16968929
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968929
parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 16968929
parsl.executors.status_handling INFO: Allocated block ID 42
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42
2024-03-09 20:54:52 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-42
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-42
2024-03-09 20:54:52 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:52 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:52 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:52 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-09 20:54:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:52 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:53 proc_lsst.multi:170 [INFO]  [multi] job_id 16968930
proc_lsst.multi INFO: [multi] job_id 16968930
2024-03-09 20:54:53 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-09 20:54:53 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:53 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16968930
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968930
parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 16968930
parsl.executors.status_handling INFO: Allocated block ID 43
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43
2024-03-09 20:54:53 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-43
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-43
2024-03-09 20:54:53 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:53 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:53 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:53 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-09 20:54:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:53 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:54 proc_lsst.multi:170 [INFO]  [multi] job_id 16968931
proc_lsst.multi INFO: [multi] job_id 16968931
2024-03-09 20:54:54 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-09 20:54:54 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:54 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16968931
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968931
parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 16968931
parsl.executors.status_handling INFO: Allocated block ID 44
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44
2024-03-09 20:54:54 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-44
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-44
2024-03-09 20:54:54 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:54 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:54 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:54 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-09 20:54:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:54 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:54 proc_lsst.multi:170 [INFO]  [multi] job_id 16968932
proc_lsst.multi INFO: [multi] job_id 16968932
2024-03-09 20:54:54 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-09 20:54:54 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:54 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16968932
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968932
parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 16968932
parsl.executors.status_handling INFO: Allocated block ID 45
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45
2024-03-09 20:54:54 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-45
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-45
2024-03-09 20:54:54 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:54 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:54 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:54 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-09 20:54:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:54 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:54 proc_lsst.multi:170 [INFO]  [multi] job_id 16968933
proc_lsst.multi INFO: [multi] job_id 16968933
2024-03-09 20:54:54 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-09 20:54:54 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:54 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16968933
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968933
parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 16968933
parsl.executors.status_handling INFO: Allocated block ID 46
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46
2024-03-09 20:54:54 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-46
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-46
2024-03-09 20:54:54 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:54 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:54 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:54 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-09 20:54:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:54 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:55 proc_lsst.multi:170 [INFO]  [multi] job_id 16968934
proc_lsst.multi INFO: [multi] job_id 16968934
2024-03-09 20:54:55 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-09 20:54:55 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:55 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16968934
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968934
parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 16968934
parsl.executors.status_handling INFO: Allocated block ID 47
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47
2024-03-09 20:54:55 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-47
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-47
2024-03-09 20:54:55 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:55 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:55 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:55 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-09 20:54:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:55 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:55 proc_lsst.multi:170 [INFO]  [multi] job_id 16968935
proc_lsst.multi INFO: [multi] job_id 16968935
2024-03-09 20:54:55 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-09 20:54:55 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:55 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16968935
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968935
parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 16968935
parsl.executors.status_handling INFO: Allocated block ID 48
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48
2024-03-09 20:54:55 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-48
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-48
2024-03-09 20:54:55 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:55 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:55 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:55 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-09 20:54:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:55 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:56 proc_lsst.multi:170 [INFO]  [multi] job_id 16968936
proc_lsst.multi INFO: [multi] job_id 16968936
2024-03-09 20:54:56 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-09 20:54:56 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:56 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16968936
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968936
parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 16968936
parsl.executors.status_handling INFO: Allocated block ID 49
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49
2024-03-09 20:54:56 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-49
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-49
2024-03-09 20:54:56 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:56 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:56 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:56 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-09 20:54:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:56 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:56 proc_lsst.multi:170 [INFO]  [multi] job_id 16968937
proc_lsst.multi INFO: [multi] job_id 16968937
2024-03-09 20:54:56 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-09 20:54:56 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:56 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16968937
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968937
parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 16968937
parsl.executors.status_handling INFO: Allocated block ID 50
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50
2024-03-09 20:54:56 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-50
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-50
2024-03-09 20:54:56 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:56 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:56 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:56 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-09 20:54:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:56 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ac4efccc3709ebb02c44e7b6e79a19e3 with result from task 9
parsl.dataflow.dflow DEBUG: Task 9 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 1fb59a25919fa162e3c017e079579c65
parsl.dataflow.memoization INFO: Task 10 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fd080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 375
parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/aee9a4a4-2cb4-4a22-8cfd-1defc1fb950e_characterizeImage_855820_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/aee9a4a4-2cb4-4a22-8cfd-1defc1fb950e_characterizeImage_855820_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/d305c9b4-bcf1-4083-b5eb-bbcb20eb33d3_isr_855820_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/d305c9b4-bcf1-4083-b5eb-bbcb20eb33d3_isr_855820_32.stderr
2024-03-09 20:54:57 proc_lsst.multi:170 [INFO]  [multi] job_id 16968938
proc_lsst.multi INFO: [multi] job_id 16968938
2024-03-09 20:54:57 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-09 20:54:57 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:57 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16968938
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968938
parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 16968938
parsl.executors.status_handling INFO: Allocated block ID 51
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51
2024-03-09 20:54:57 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-51
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-51
2024-03-09 20:54:57 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:57 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:57 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:57 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-09 20:54:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:57 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:57 proc_lsst.multi:170 [INFO]  [multi] job_id 16968939
proc_lsst.multi INFO: [multi] job_id 16968939
2024-03-09 20:54:57 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-09 20:54:57 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:57 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16968939
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968939
parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 16968939
parsl.executors.status_handling INFO: Allocated block ID 52
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52
2024-03-09 20:54:57 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-52
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-52
2024-03-09 20:54:57 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:57 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:57 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:57 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-09 20:54:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:57 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:57 proc_lsst.multi:170 [INFO]  [multi] job_id 16968940
proc_lsst.multi INFO: [multi] job_id 16968940
2024-03-09 20:54:57 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-09 20:54:57 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:57 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16968940
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968940
parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 16968940
parsl.executors.status_handling INFO: Allocated block ID 53
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53
2024-03-09 20:54:57 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-53
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-53
2024-03-09 20:54:57 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:57 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:57 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:57 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-09 20:54:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:57 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:58 proc_lsst.multi:170 [INFO]  [multi] job_id 16968941
proc_lsst.multi INFO: [multi] job_id 16968941
2024-03-09 20:54:58 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-09 20:54:58 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:58 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16968941
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968941
parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 16968941
parsl.executors.status_handling INFO: Allocated block ID 54
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54
2024-03-09 20:54:58 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-54
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-54
2024-03-09 20:54:58 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:58 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:58 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:58 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-09 20:54:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:58 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:58 proc_lsst.multi:170 [INFO]  [multi] job_id 16968942
proc_lsst.multi INFO: [multi] job_id 16968942
2024-03-09 20:54:58 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-09 20:54:58 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:58 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16968942
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968942
parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 16968942
parsl.executors.status_handling INFO: Allocated block ID 55
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55
2024-03-09 20:54:58 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-55
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-55
2024-03-09 20:54:58 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:58 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:58 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:58 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-09 20:54:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:58 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:58 proc_lsst.multi:170 [INFO]  [multi] job_id 16968943
proc_lsst.multi INFO: [multi] job_id 16968943
2024-03-09 20:54:58 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:54:58 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:58 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16968943
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968943
parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 16968943
parsl.executors.status_handling INFO: Allocated block ID 56
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56
2024-03-09 20:54:58 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-56
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-56
2024-03-09 20:54:58 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:58 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:58 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:58 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:54:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:58 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-09 20:54:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:54:58 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:59 proc_lsst.multi:170 [INFO]  [multi] job_id 16968944
proc_lsst.multi INFO: [multi] job_id 16968944
2024-03-09 20:54:59 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:59 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:54:59 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968944
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968944
parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 16968944
parsl.executors.status_handling INFO: Allocated block ID 57
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57
2024-03-09 20:54:59 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-57
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-57
2024-03-09 20:54:59 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:59 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:59 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:59 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:54:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:59 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:54:59 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:54:59 proc_lsst.multi:170 [INFO]  [multi] job_id 16968945
proc_lsst.multi INFO: [multi] job_id 16968945
2024-03-09 20:54:59 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-09 20:54:59 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:54:59 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968945
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968945
parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 16968945
parsl.executors.status_handling INFO: Allocated block ID 58
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58
2024-03-09 20:54:59 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-58
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-58
2024-03-09 20:54:59 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:54:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:54:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:54:59 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:54:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:54:59 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:54:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:54:59 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:54:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:54:59 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:54:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-09 20:54:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:54:59 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:55:00 proc_lsst.multi:170 [INFO]  [multi] job_id 16968946
proc_lsst.multi INFO: [multi] job_id 16968946
2024-03-09 20:55:00 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-09 20:55:00 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:00 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968946
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968946
parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 16968946
parsl.executors.status_handling INFO: Allocated block ID 59
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59
2024-03-09 20:55:00 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-59
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-59
2024-03-09 20:55:00 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:00 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:00 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:00 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:00 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-09 20:55:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:00 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:55:00 proc_lsst.multi:170 [INFO]  [multi] job_id 16968947
proc_lsst.multi INFO: [multi] job_id 16968947
2024-03-09 20:55:00 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-09 20:55:00 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:00 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968947
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968947
parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 16968947
parsl.executors.status_handling INFO: Allocated block ID 60
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60
2024-03-09 20:55:00 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-60
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-60
2024-03-09 20:55:00 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:00 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:00 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:00 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:00 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-09 20:55:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:00 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:55:01 proc_lsst.multi:170 [INFO]  [multi] job_id 16968948
proc_lsst.multi INFO: [multi] job_id 16968948
2024-03-09 20:55:01 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:01 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:01 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968948
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968948
parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 16968948
parsl.executors.status_handling INFO: Allocated block ID 61
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61
2024-03-09 20:55:01 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-61
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-61
2024-03-09 20:55:01 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:01 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:01 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:01 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:01 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:01 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bc0c11569bb434b1197674500359d246 with result from task 12
parsl.dataflow.dflow DEBUG: Task 12 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 0c7064d8d70118060abc4d39fafa6f00
parsl.dataflow.memoization INFO: Task 13 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fd6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 376
parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/624cec9e-6cce-47d8-ad03-90deacaefc63_characterizeImage_855911_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/624cec9e-6cce-47d8-ad03-90deacaefc63_characterizeImage_855911_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/9f963235-ea2e-4b32-b719-0d08d19abe6a_isr_855911_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/9f963235-ea2e-4b32-b719-0d08d19abe6a_isr_855911_57.stderr
2024-03-09 20:55:02 proc_lsst.multi:170 [INFO]  [multi] job_id 16968949
proc_lsst.multi INFO: [multi] job_id 16968949
2024-03-09 20:55:02 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-09 20:55:02 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:02 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968949
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968949
parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 16968949
parsl.executors.status_handling INFO: Allocated block ID 62
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62
2024-03-09 20:55:02 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-62
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-62
2024-03-09 20:55:02 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:02 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:02 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:02 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:02 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-09 20:55:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:02 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d41eaa319e0976cbd403945f6ef1ac82 with result from task 6
parsl.dataflow.dflow DEBUG: Task 6 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 7 has memoization hash ae988a5419b2c436fd4bf2e18effe8fe
parsl.dataflow.memoization INFO: Task 7 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fcc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 377
parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/1ee2d914-0cde-4dee-b6a3-167da531173f_characterizeImage_855822_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/1ee2d914-0cde-4dee-b6a3-167da531173f_characterizeImage_855822_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/0e6c775b-f198-4540-b7ca-6afc65c32e13_isr_855822_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/0e6c775b-f198-4540-b7ca-6afc65c32e13_isr_855822_13.stderr
2024-03-09 20:55:02 proc_lsst.multi:170 [INFO]  [multi] job_id 16968950
proc_lsst.multi INFO: [multi] job_id 16968950
2024-03-09 20:55:02 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-09 20:55:02 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:02 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968950
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968950
parsl.executors.status_handling DEBUG: Launched block 62 on executor multi with job ID 16968950
parsl.executors.status_handling INFO: Allocated block ID 63
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63
2024-03-09 20:55:02 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-63
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-63
2024-03-09 20:55:02 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:02 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:02 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:02 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:02 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-09 20:55:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:02 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:55:02 proc_lsst.multi:170 [INFO]  [multi] job_id 16968951
proc_lsst.multi INFO: [multi] job_id 16968951
2024-03-09 20:55:02 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-09 20:55:02 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:02 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968951
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968951
parsl.executors.status_handling DEBUG: Launched block 63 on executor multi with job ID 16968951
parsl.executors.status_handling INFO: Allocated block ID 64
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64
2024-03-09 20:55:02 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-64
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-64
2024-03-09 20:55:02 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:02 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:02 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:02 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:02 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-09 20:55:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:02 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:55:03 proc_lsst.multi:170 [INFO]  [multi] job_id 16968952
proc_lsst.multi INFO: [multi] job_id 16968952
2024-03-09 20:55:03 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-09 20:55:03 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:03 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968952
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968952
parsl.executors.status_handling DEBUG: Launched block 64 on executor multi with job ID 16968952
parsl.executors.status_handling INFO: Allocated block ID 65
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65
2024-03-09 20:55:03 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-65
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-65
2024-03-09 20:55:03 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:03 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:03 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:03 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:03 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-09 20:55:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:03 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:55:04 proc_lsst.multi:170 [INFO]  [multi] job_id 16968953
proc_lsst.multi INFO: [multi] job_id 16968953
2024-03-09 20:55:04 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-09 20:55:04 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:04 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968953
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968953
parsl.executors.status_handling DEBUG: Launched block 65 on executor multi with job ID 16968953
parsl.executors.status_handling INFO: Allocated block ID 66
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66
2024-03-09 20:55:04 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-66
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-66
2024-03-09 20:55:04 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:04 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:04 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:04 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:04 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-09 20:55:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:04 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:55:04 proc_lsst.multi:170 [INFO]  [multi] job_id 16968954
proc_lsst.multi INFO: [multi] job_id 16968954
2024-03-09 20:55:04 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-09 20:55:04 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:04 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968954
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968954
parsl.executors.status_handling DEBUG: Launched block 66 on executor multi with job ID 16968954
parsl.executors.status_handling INFO: Allocated block ID 67
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67
2024-03-09 20:55:04 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-67
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-67
2024-03-09 20:55:04 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:04 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:04 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:04 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:04 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-09 20:55:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:04 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 40285742b27a80ae6e5b2c601c510166 with result from task 75
parsl.dataflow.dflow DEBUG: Task 75 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 6d86c1738ee5ffe58ba6fffb442a5fea
parsl.dataflow.memoization INFO: Task 76 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 378
parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a7ba73de-60e6-4bf8-a7dd-e884c059c8dd_characterizeImage_855822_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a7ba73de-60e6-4bf8-a7dd-e884c059c8dd_characterizeImage_855822_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/47943823-dfae-49b6-8a8e-6fbc789442fb_isr_855822_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/47943823-dfae-49b6-8a8e-6fbc789442fb_isr_855822_2.stderr
2024-03-09 20:55:05 proc_lsst.multi:170 [INFO]  [multi] job_id 16968955
proc_lsst.multi INFO: [multi] job_id 16968955
2024-03-09 20:55:05 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-09 20:55:05 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:05 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968955
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968955
parsl.executors.status_handling DEBUG: Launched block 67 on executor multi with job ID 16968955
parsl.executors.status_handling INFO: Allocated block ID 68
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68
2024-03-09 20:55:05 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-68
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-68
2024-03-09 20:55:05 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:05 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:05 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:05 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:05 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-09 20:55:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:05 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:55:05 proc_lsst.multi:170 [INFO]  [multi] job_id 16968956
proc_lsst.multi INFO: [multi] job_id 16968956
2024-03-09 20:55:05 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-09 20:55:05 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:05 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968956
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968956
parsl.executors.status_handling DEBUG: Launched block 68 on executor multi with job ID 16968956
parsl.executors.status_handling INFO: Allocated block ID 69
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69
2024-03-09 20:55:05 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-69
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-69
2024-03-09 20:55:05 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:05 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:05 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:05 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:05 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-09 20:55:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:05 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry feb930155cd1ac4465090205fe48e9d6 with result from task 66
parsl.dataflow.dflow DEBUG: Task 66 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 72dafb5d26fd924d15ba7296be976376
parsl.dataflow.memoization INFO: Task 67 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 379
parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/e48d4589-e0c5-4812-8dcb-94e46c3bd2fa_characterizeImage_855909_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/e48d4589-e0c5-4812-8dcb-94e46c3bd2fa_characterizeImage_855909_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/e25e2421-b5cd-476a-93cb-1c478625d519_isr_855909_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/e25e2421-b5cd-476a-93cb-1c478625d519_isr_855909_2.stderr
2024-03-09 20:55:06 proc_lsst.multi:170 [INFO]  [multi] job_id 16968957
proc_lsst.multi INFO: [multi] job_id 16968957
2024-03-09 20:55:06 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-09 20:55:06 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:06 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968957
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968957
parsl.executors.status_handling DEBUG: Launched block 69 on executor multi with job ID 16968957
parsl.executors.status_handling INFO: Allocated block ID 70
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70
2024-03-09 20:55:06 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-70
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-70
2024-03-09 20:55:06 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:06 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:06 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:06 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:06 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-09 20:55:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:06 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:55:06 proc_lsst.multi:170 [INFO]  [multi] job_id 16968958
proc_lsst.multi INFO: [multi] job_id 16968958
2024-03-09 20:55:06 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-09 20:55:06 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:06 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968958
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968958
parsl.executors.status_handling DEBUG: Launched block 70 on executor multi with job ID 16968958
parsl.executors.status_handling INFO: Allocated block ID 71
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71
2024-03-09 20:55:06 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-71
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-71
2024-03-09 20:55:06 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:06 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:06 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:06 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:06 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-09 20:55:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:06 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d085d2350d706208e3cd79a536db4f78 with result from task 39
parsl.dataflow.dflow DEBUG: Task 39 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 40 has memoization hash 256c2ecc325be0ea77b659422435d5b9
parsl.dataflow.memoization INFO: Task 40 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fde40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 380
parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/70f5ed46-83e2-42a4-91f4-254084eab382_characterizeImage_855909_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/70f5ed46-83e2-42a4-91f4-254084eab382_characterizeImage_855909_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/3732fe83-9831-4d50-9663-71cf5b0455cc_isr_855909_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/3732fe83-9831-4d50-9663-71cf5b0455cc_isr_855909_31.stderr
parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7348f312bacb63cecb5b879cb5c1971c with result from task 30
parsl.dataflow.dflow DEBUG: Task 30 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 31 has memoization hash d4bb03926925e669bb7f6b32f6271d82
parsl.dataflow.memoization INFO: Task 31 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fe8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 381
parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/6bc69d38-1305-453b-a53c-54929b7e9534_characterizeImage_855822_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/6bc69d38-1305-453b-a53c-54929b7e9534_characterizeImage_855822_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/156e09da-e1d4-45e1-acb5-2a19ece72141_isr_855822_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/156e09da-e1d4-45e1-acb5-2a19ece72141_isr_855822_8.stderr
parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 22b712c2bd9ec854036f1310ca03ce70 with result from task 36
parsl.dataflow.dflow DEBUG: Task 36 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 4acb85c886b42beb887a9239dd63a178
parsl.dataflow.memoization INFO: Task 37 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fe2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 382
parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/00256e21-f162-48b0-abac-1faad383dbc9_characterizeImage_855909_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/00256e21-f162-48b0-abac-1faad383dbc9_characterizeImage_855909_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/faf088f9-84e9-4189-92b4-169f97e6ebce_isr_855909_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/faf088f9-84e9-4189-92b4-169f97e6ebce_isr_855909_34.stderr
2024-03-09 20:55:07 proc_lsst.multi:170 [INFO]  [multi] job_id 16968959
proc_lsst.multi INFO: [multi] job_id 16968959
2024-03-09 20:55:07 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-09 20:55:07 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:07 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968959
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968959
parsl.executors.status_handling DEBUG: Launched block 71 on executor multi with job ID 16968959
parsl.executors.status_handling INFO: Allocated block ID 72
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72
2024-03-09 20:55:07 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-72
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-72
2024-03-09 20:55:07 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:07 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:07 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:07 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:07 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-09 20:55:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:07 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9f43ee98fa484ca34a545342548830d0 with result from task 24
parsl.dataflow.dflow DEBUG: Task 24 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 883720ff63528a288330fca463aecd47
parsl.dataflow.memoization INFO: Task 25 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 383
parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/fc869b7e-88d6-4c01-b3d5-35d5d46b07ad_characterizeImage_855822_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/fc869b7e-88d6-4c01-b3d5-35d5d46b07ad_characterizeImage_855822_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/25977ed8-d795-46e2-8f46-894a614d9923_isr_855822_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/25977ed8-d795-46e2-8f46-894a614d9923_isr_855822_9.stderr
parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 744eb2631fb9fa64539d15a454d622ca with result from task 48
parsl.dataflow.dflow DEBUG: Task 48 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 6f1fe045f887b8a4f9535a310134a3b3
parsl.dataflow.memoization INFO: Task 49 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169ffce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 384
parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/fb9b9416-82e6-4ec8-aa76-4226497178bc_characterizeImage_855909_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/fb9b9416-82e6-4ec8-aa76-4226497178bc_characterizeImage_855909_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/8e01a36d-2979-4279-bb83-9fc0f1867c8e_isr_855909_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/8e01a36d-2979-4279-bb83-9fc0f1867c8e_isr_855909_39.stderr
parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2771011aa5692236f5b92c631eeab51b with result from task 63
parsl.dataflow.dflow DEBUG: Task 63 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 64 has memoization hash fe04ec56a4904e1a32e6629d91ec2d04
parsl.dataflow.memoization INFO: Task 64 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 385
parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/0e5cfb16-d2c2-488a-8796-94d135ecc930_characterizeImage_855822_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/0e5cfb16-d2c2-488a-8796-94d135ecc930_characterizeImage_855822_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/a3aae6ae-afac-4129-82f6-409b9c9f6719_isr_855822_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/a3aae6ae-afac-4129-82f6-409b9c9f6719_isr_855822_11.stderr
parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b2e47e9b77b6a330505ac0af31e84c92 with result from task 15
parsl.dataflow.dflow DEBUG: Task 15 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 47554c015f0ad7bb8482c47850623974
parsl.dataflow.memoization INFO: Task 16 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 386
parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/730f3db9-d95d-4bb9-b528-26acafd80720_characterizeImage_855909_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/730f3db9-d95d-4bb9-b528-26acafd80720_characterizeImage_855909_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/fa913612-5e28-4c5f-9c8f-e8719d5c54ed_isr_855909_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/fa913612-5e28-4c5f-9c8f-e8719d5c54ed_isr_855909_45.stderr
parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7eb106b1da7db62332917c20eec76b4e with result from task 33
parsl.dataflow.dflow DEBUG: Task 33 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 28bf0f55dd02f7b245dc5874e8169997
parsl.dataflow.memoization INFO: Task 34 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fe5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 387
parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/5d3158c4-6eb6-4ef8-b1ee-c628bc43e0c5_characterizeImage_855910_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/5d3158c4-6eb6-4ef8-b1ee-c628bc43e0c5_characterizeImage_855910_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/4b845abe-c313-4d31-b447-e0151279eeb5_isr_855910_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/4b845abe-c313-4d31-b447-e0151279eeb5_isr_855910_56.stderr
parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2a38537c09a964e6c2831af6714ad131 with result from task 45
parsl.dataflow.dflow DEBUG: Task 45 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 0679a9f8ced9b8db612a67ddfd7cd052
parsl.dataflow.memoization INFO: Task 46 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169ff9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 388
parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/bbe7210e-b3fc-451c-9ae0-52ec4198fdeb_characterizeImage_855820_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/bbe7210e-b3fc-451c-9ae0-52ec4198fdeb_characterizeImage_855820_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/ced7aad1-1d5c-4875-bd34-f81a0372b49c_isr_855820_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/ced7aad1-1d5c-4875-bd34-f81a0372b49c_isr_855820_27.stderr
parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2183a36d7c59cf4e408f2de6dad43297 with result from task 27
parsl.dataflow.dflow DEBUG: Task 27 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 28 has memoization hash d65319fdaf240d1fd266d4fdc7251656
parsl.dataflow.memoization INFO: Task 28 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 389
parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/26eb6126-3dec-4e32-89ba-2f15a60704bb_characterizeImage_855910_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/26eb6126-3dec-4e32-89ba-2f15a60704bb_characterizeImage_855910_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/e34aad39-a850-4d6a-82cc-4a92ef347cc3_isr_855910_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/e34aad39-a850-4d6a-82cc-4a92ef347cc3_isr_855910_1.stderr
parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b7fa37d7e40f4e14ad4a246b6f43a394 with result from task 60
parsl.dataflow.dflow DEBUG: Task 60 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 671678e604c26484b591d5d13831871e
parsl.dataflow.memoization INFO: Task 61 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 390
parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/fd018433-8f13-4185-8dd5-1c952533989d_characterizeImage_855910_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/fd018433-8f13-4185-8dd5-1c952533989d_characterizeImage_855910_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/35bd3de8-b48a-4883-bddd-96bf771c4460_isr_855910_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/35bd3de8-b48a-4883-bddd-96bf771c4460_isr_855910_20.stderr
2024-03-09 20:55:07 proc_lsst.multi:170 [INFO]  [multi] job_id 16968960
proc_lsst.multi INFO: [multi] job_id 16968960
2024-03-09 20:55:07 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-09 20:55:07 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:07 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968960
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968960
parsl.executors.status_handling DEBUG: Launched block 72 on executor multi with job ID 16968960
parsl.executors.status_handling INFO: Allocated block ID 73
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73
2024-03-09 20:55:07 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-73
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-73
2024-03-09 20:55:07 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:07 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:07 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:07 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:07 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-09 20:55:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:07 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aee033a5e0a883b2acdad73f9713644d with result from task 72
parsl.dataflow.dflow DEBUG: Task 72 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 250d620d1601e63d735774a717cd6274
parsl.dataflow.memoization INFO: Task 73 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 391
parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/54458e78-9027-47ce-862b-4d889c61ef6f_characterizeImage_855910_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/54458e78-9027-47ce-862b-4d889c61ef6f_characterizeImage_855910_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/6a707045-5237-4f9d-84ee-d5dc47fc1ba4_isr_855910_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/6a707045-5237-4f9d-84ee-d5dc47fc1ba4_isr_855910_45.stderr
parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 211830d6fe1a3251251b25d48b0ddec1 with result from task 57
parsl.dataflow.dflow DEBUG: Task 57 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 09f402290d993f848af5644b97bee408
parsl.dataflow.memoization INFO: Task 58 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 392
parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/6bee672c-7f28-438b-a7b2-fd625a1c2f5b_characterizeImage_855822_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/6bee672c-7f28-438b-a7b2-fd625a1c2f5b_characterizeImage_855822_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/da116374-d7ef-4458-914d-0218b259358e_isr_855822_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/da116374-d7ef-4458-914d-0218b259358e_isr_855822_41.stderr
parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 80770de630fe4d2f4fbc5e9e36a58e13 with result from task 18
parsl.dataflow.dflow DEBUG: Task 18 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 84c5171cde042008813819702c604184
parsl.dataflow.memoization INFO: Task 19 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 393
parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/b9dcbc31-bbba-4201-a5ca-aa76b4064b54_characterizeImage_855822_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/b9dcbc31-bbba-4201-a5ca-aa76b4064b54_characterizeImage_855822_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/c61aa638-084e-4483-91ab-8fbcdc3923e1_isr_855822_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/c61aa638-084e-4483-91ab-8fbcdc3923e1_isr_855822_10.stderr
parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b35afb0315bf991d39f3a41dd9ce12ff with result from task 87
parsl.dataflow.dflow DEBUG: Task 87 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 88 has memoization hash ec3fb2162981d5b858eb24e266ebcc01
parsl.dataflow.memoization INFO: Task 88 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 394
parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/5daa879c-b376-49e2-a408-eebd8244eeeb_characterizeImage_855820_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/5daa879c-b376-49e2-a408-eebd8244eeeb_characterizeImage_855820_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/ed4b9191-ad41-47b6-a2ba-c0bea4275aea_isr_855820_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/ed4b9191-ad41-47b6-a2ba-c0bea4275aea_isr_855820_43.stderr
parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0e3165c7e18be337269c97761a623f7a with result from task 21
parsl.dataflow.dflow DEBUG: Task 21 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 823b20bd90d4590b1704b4eb8dfd57e2
parsl.dataflow.memoization INFO: Task 22 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169ff1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 395
parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/48534c7f-0fc6-4cab-8651-2ba90640d703_characterizeImage_855909_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/48534c7f-0fc6-4cab-8651-2ba90640d703_characterizeImage_855909_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/6e0135bb-233d-40e6-b768-b0e91a61e8df_isr_855909_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/6e0135bb-233d-40e6-b768-b0e91a61e8df_isr_855909_4.stderr
parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5806a75423fe6c7f67dc74d3df2c78ae with result from task 54
parsl.dataflow.dflow DEBUG: Task 54 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 2ae285dffcef1e0e82b3bb35f8a3a3cf
parsl.dataflow.memoization INFO: Task 55 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 396
parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/197d3d62-39c7-4303-b5c4-6f03c71107cb_characterizeImage_855911_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/197d3d62-39c7-4303-b5c4-6f03c71107cb_characterizeImage_855911_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/b0495512-328b-4c97-8df6-f9486db34b9e_isr_855911_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/b0495512-328b-4c97-8df6-f9486db34b9e_isr_855911_22.stderr
parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ccb59a432459a7ae58d6335dbb330d16 with result from task 78
parsl.dataflow.dflow DEBUG: Task 78 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 64d3abfd5a280f72b53916f973f500e4
parsl.dataflow.memoization INFO: Task 79 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 397
parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/41865dfc-30a2-44dd-ac08-23a65c066bbf_characterizeImage_855821_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/41865dfc-30a2-44dd-ac08-23a65c066bbf_characterizeImage_855821_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/b5255453-23d4-4325-b5f6-75509c071f6e_isr_855821_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/b5255453-23d4-4325-b5f6-75509c071f6e_isr_855821_16.stderr
parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 96d36506deda9389e18fc6a892ef9dac with result from task 81
parsl.dataflow.dflow DEBUG: Task 81 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 540122b77642909a697289944dd95cf3
parsl.dataflow.memoization INFO: Task 82 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 398
parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/1414d790-0808-4fb3-a913-436ff16a524d_characterizeImage_855820_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/1414d790-0808-4fb3-a913-436ff16a524d_characterizeImage_855820_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/96ad5f4e-d5cb-4304-8f0f-9dce5ac62085_isr_855820_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/96ad5f4e-d5cb-4304-8f0f-9dce5ac62085_isr_855820_11.stderr
2024-03-09 20:55:08 proc_lsst.multi:170 [INFO]  [multi] job_id 16968961
proc_lsst.multi INFO: [multi] job_id 16968961
2024-03-09 20:55:08 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-09 20:55:08 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:08 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968961
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968961
parsl.executors.status_handling DEBUG: Launched block 73 on executor multi with job ID 16968961
parsl.executors.status_handling INFO: Allocated block ID 74
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74
2024-03-09 20:55:08 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-74
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-74
2024-03-09 20:55:08 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:08 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:08 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:08 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:08 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-09 20:55:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:08 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 60d178f14c015bbbf25164278e20b920 with result from task 51
parsl.dataflow.dflow DEBUG: Task 51 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 082b65c7a03299ea4bfed63004b61177
parsl.dataflow.memoization INFO: Task 52 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169ffe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 399
parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/7fed83ac-fb51-430a-b554-a0d20f137966_characterizeImage_855822_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/7fed83ac-fb51-430a-b554-a0d20f137966_characterizeImage_855822_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/218562ee-24ca-4966-99fd-3265df808d47_isr_855822_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/218562ee-24ca-4966-99fd-3265df808d47_isr_855822_48.stderr
2024-03-09 20:55:09 proc_lsst.multi:170 [INFO]  [multi] job_id 16968962
proc_lsst.multi INFO: [multi] job_id 16968962
2024-03-09 20:55:09 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-09 20:55:09 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:09 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968962
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968962
parsl.executors.status_handling DEBUG: Launched block 74 on executor multi with job ID 16968962
parsl.executors.status_handling INFO: Allocated block ID 75
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75
2024-03-09 20:55:09 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-75
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-75
2024-03-09 20:55:09 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:09 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:09 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:09 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:09 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-09 20:55:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:09 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a87423074a1d5bf78b40171ec326329a with result from task 84
parsl.dataflow.dflow DEBUG: Task 84 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 85 has memoization hash a754a308a897ac88bcf88cfd7690d033
parsl.dataflow.memoization INFO: Task 85 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 400
parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/e3fa9959-4fc0-43f8-8bca-2cb22c80a080_characterizeImage_855821_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/e3fa9959-4fc0-43f8-8bca-2cb22c80a080_characterizeImage_855821_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/7ee11a91-1eef-40ee-8030-6528934a4944_isr_855821_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/7ee11a91-1eef-40ee-8030-6528934a4944_isr_855821_12.stderr
2024-03-09 20:55:09 proc_lsst.multi:170 [INFO]  [multi] job_id 16968963
proc_lsst.multi INFO: [multi] job_id 16968963
2024-03-09 20:55:09 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:09 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:09 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968963
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968963
parsl.executors.status_handling DEBUG: Launched block 75 on executor multi with job ID 16968963
parsl.executors.status_handling INFO: Allocated block ID 76
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76
2024-03-09 20:55:09 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-76
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-76
2024-03-09 20:55:09 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:09 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:09 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:09 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:09 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:09 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 09746aa78871951a55bfa3265a23be31 with result from task 69
parsl.dataflow.dflow DEBUG: Task 69 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 70 has memoization hash e50a8d9251565a8be5792ba9cce50354
parsl.dataflow.memoization INFO: Task 70 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 401
parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/782dbd87-3469-4694-b922-6b1a5405e0ab_characterizeImage_855909_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/782dbd87-3469-4694-b922-6b1a5405e0ab_characterizeImage_855909_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/5b572b29-b459-4dd5-b65a-824f2254a1e3_isr_855909_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/5b572b29-b459-4dd5-b65a-824f2254a1e3_isr_855909_37.stderr
parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9d8bd4496964db03e555fe85d7d531eb with result from task 90
parsl.dataflow.dflow DEBUG: Task 90 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 7cfdb8d27b1724859b2add5dc7bbc51b
parsl.dataflow.memoization INFO: Task 91 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 402
parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/72a0c7e7-fa1a-4a3f-b3a4-ae297767711c_characterizeImage_855910_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/72a0c7e7-fa1a-4a3f-b3a4-ae297767711c_characterizeImage_855910_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/0d62d0fc-d715-4efa-885d-1081b0e5810c_isr_855910_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/0d62d0fc-d715-4efa-885d-1081b0e5810c_isr_855910_37.stderr
parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4934caed02d9ba35e683642c8da84bcc with result from task 42
parsl.dataflow.dflow DEBUG: Task 42 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 7ac38e7d30256d1c7605bfe53f6e8eb9
parsl.dataflow.memoization INFO: Task 43 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169ff6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 403
parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/7c0cdb32-7c95-4250-a744-924d7e612461_characterizeImage_855820_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/7c0cdb32-7c95-4250-a744-924d7e612461_characterizeImage_855820_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/a2bfdb1d-931c-4f98-9961-abfb57a4737b_isr_855820_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/a2bfdb1d-931c-4f98-9961-abfb57a4737b_isr_855820_58.stderr
2024-03-09 20:55:10 proc_lsst.multi:170 [INFO]  [multi] job_id 16968964
proc_lsst.multi INFO: [multi] job_id 16968964
2024-03-09 20:55:10 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-09 20:55:10 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:10 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968964
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968964
parsl.executors.status_handling DEBUG: Launched block 76 on executor multi with job ID 16968964
parsl.executors.status_handling INFO: Allocated block ID 77
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77
2024-03-09 20:55:10 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-77
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-77
2024-03-09 20:55:10 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:10 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:10 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:10 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:10 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-09 20:55:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:10 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:55:10 proc_lsst.multi:170 [INFO]  [multi] job_id 16968965
proc_lsst.multi INFO: [multi] job_id 16968965
2024-03-09 20:55:10 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-09 20:55:10 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:10 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968965
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968965
parsl.executors.status_handling DEBUG: Launched block 77 on executor multi with job ID 16968965
parsl.executors.status_handling INFO: Allocated block ID 78
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-78
2024-03-09 20:55:10 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-78
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-78
2024-03-09 20:55:10 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:10 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:10 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:10 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:10 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-09 20:55:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:10 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:55:11 proc_lsst.multi:170 [INFO]  [multi] job_id 16968966
proc_lsst.multi INFO: [multi] job_id 16968966
2024-03-09 20:55:11 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-09 20:55:11 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:11 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968966
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968966
parsl.executors.status_handling DEBUG: Launched block 78 on executor multi with job ID 16968966
parsl.executors.status_handling INFO: Allocated block ID 79
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-79
2024-03-09 20:55:11 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-79
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-79
2024-03-09 20:55:11 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:11 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:11 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:11 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:11 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-09 20:55:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:11 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 353abf2401e115a909a5d82313bf99b2 with result from task 93
parsl.dataflow.dflow DEBUG: Task 93 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 88f36cd4e3ce63fd6ff9070d5ea2e6fb
parsl.dataflow.memoization INFO: Task 94 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 404
parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/6af7edc2-50de-4381-9d68-e22864275a80_characterizeImage_855910_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/6af7edc2-50de-4381-9d68-e22864275a80_characterizeImage_855910_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/334a2251-3dbb-4c74-b6a5-3ce5cd77ad38_isr_855910_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/334a2251-3dbb-4c74-b6a5-3ce5cd77ad38_isr_855910_55.stderr
2024-03-09 20:55:11 proc_lsst.multi:170 [INFO]  [multi] job_id 16968967
proc_lsst.multi INFO: [multi] job_id 16968967
2024-03-09 20:55:11 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-09 20:55:11 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:11 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968967
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968967
parsl.executors.status_handling DEBUG: Launched block 79 on executor multi with job ID 16968967
parsl.executors.status_handling INFO: Allocated block ID 80
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-80
2024-03-09 20:55:11 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-80
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-80
2024-03-09 20:55:11 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:11 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:11 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:11 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:11 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-09 20:55:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:11 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:55:11 proc_lsst.multi:170 [INFO]  [multi] job_id 16968968
proc_lsst.multi INFO: [multi] job_id 16968968
2024-03-09 20:55:11 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-09 20:55:11 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:11 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968968
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968968
parsl.executors.status_handling DEBUG: Launched block 80 on executor multi with job ID 16968968
parsl.executors.status_handling INFO: Allocated block ID 81
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-81
2024-03-09 20:55:11 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-81
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-81
2024-03-09 20:55:11 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:11 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:11 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:11 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:11 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-09 20:55:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:11 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:55:12 proc_lsst.multi:170 [INFO]  [multi] job_id 16968969
proc_lsst.multi INFO: [multi] job_id 16968969
2024-03-09 20:55:12 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-09 20:55:12 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:12 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968969
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968969
parsl.executors.status_handling DEBUG: Launched block 81 on executor multi with job ID 16968969
parsl.executors.status_handling INFO: Allocated block ID 82
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-82
2024-03-09 20:55:12 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-82
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-82
2024-03-09 20:55:12 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:12 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:12 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:12 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:12 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-09 20:55:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:12 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:55:12 proc_lsst.multi:170 [INFO]  [multi] job_id 16968970
proc_lsst.multi INFO: [multi] job_id 16968970
2024-03-09 20:55:12 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-09 20:55:12 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:12 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968970
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968970
parsl.executors.status_handling DEBUG: Launched block 82 on executor multi with job ID 16968970
parsl.executors.status_handling INFO: Allocated block ID 83
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-83
2024-03-09 20:55:12 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-83
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-83
2024-03-09 20:55:12 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:12 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:12 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:12 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:12 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-09 20:55:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:12 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:55:13 proc_lsst.multi:170 [INFO]  [multi] job_id 16968971
proc_lsst.multi INFO: [multi] job_id 16968971
2024-03-09 20:55:13 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-09 20:55:13 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:13 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968971
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968971
parsl.executors.status_handling DEBUG: Launched block 83 on executor multi with job ID 16968971
parsl.executors.status_handling INFO: Allocated block ID 84
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-84
2024-03-09 20:55:13 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-84
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-84
2024-03-09 20:55:13 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:13 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:13 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:13 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:13 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:13 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:13 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:13 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:13 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:13 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:13 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:13 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:13 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:13 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-09 20:55:13 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:13 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:55:13 proc_lsst.multi:170 [INFO]  [multi] job_id 16968972
proc_lsst.multi INFO: [multi] job_id 16968972
2024-03-09 20:55:13 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-09 20:55:13 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:13 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968972
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968972
parsl.executors.status_handling DEBUG: Launched block 84 on executor multi with job ID 16968972
parsl.executors.status_handling INFO: Allocated block ID 85
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-85
2024-03-09 20:55:13 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-85
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-85
2024-03-09 20:55:13 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:13 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:13 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:13 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:13 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:13 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:13 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:13 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:13 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:13 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:13 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:13 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:13 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:13 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-09 20:55:13 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:13 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:55:13 proc_lsst.multi:170 [INFO]  [multi] job_id 16968973
proc_lsst.multi INFO: [multi] job_id 16968973
2024-03-09 20:55:13 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:13 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:13 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968973
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968973
parsl.executors.status_handling DEBUG: Launched block 85 on executor multi with job ID 16968973
parsl.executors.status_handling INFO: Allocated block ID 86
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-86
2024-03-09 20:55:13 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-86
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-86
2024-03-09 20:55:13 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:13 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:13 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:13 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:13 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:13 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:13 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:13 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:13 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:13 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:13 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:13 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:13 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:13 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:13 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:13 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3d2f04fe1e4a9a55c43a0223c93a5c8e with result from task 105
parsl.dataflow.dflow DEBUG: Task 105 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 0a83697838542b4dce18be5a90fb11ca
parsl.dataflow.memoization INFO: Task 106 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 405
parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/e19b6db5-756d-4b82-88ce-7955999a4c81_characterizeImage_855909_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/e19b6db5-756d-4b82-88ce-7955999a4c81_characterizeImage_855909_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/b2053b0c-bd83-41fe-91b6-7896895a98df_isr_855909_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/b2053b0c-bd83-41fe-91b6-7896895a98df_isr_855909_49.stderr
2024-03-09 20:55:14 proc_lsst.multi:170 [INFO]  [multi] job_id 16968974
proc_lsst.multi INFO: [multi] job_id 16968974
2024-03-09 20:55:14 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 31
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31
2024-03-09 20:55:14 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:14 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968974
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968974
parsl.executors.status_handling DEBUG: Launched block 86 on executor multi with job ID 16968974
parsl.executors.status_handling INFO: Allocated block ID 87
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-87
2024-03-09 20:55:14 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-87
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-87
2024-03-09 20:55:14 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:14 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:14 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:14 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:14 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 31
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31
2024-03-09 20:55:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:14 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:55:14 proc_lsst.multi:170 [INFO]  [multi] job_id 16968975
proc_lsst.multi INFO: [multi] job_id 16968975
2024-03-09 20:55:14 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 32
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32
2024-03-09 20:55:14 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:14 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968975
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968975
parsl.executors.status_handling DEBUG: Launched block 87 on executor multi with job ID 16968975
parsl.executors.status_handling INFO: Allocated block ID 88
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-88
2024-03-09 20:55:14 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-88
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-88
2024-03-09 20:55:14 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:14 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:14 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:14 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:14 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 32
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32
2024-03-09 20:55:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:14 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 84cf7b6b62eec8696032bcd74c7eae02 with result from task 108
parsl.dataflow.dflow DEBUG: Task 108 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 66f83491c8abdfa9ce0e9ae385d8654a
parsl.dataflow.memoization INFO: Task 109 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 406
parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/d62fb751-c5bb-4036-b9e0-ba333f7ef680_characterizeImage_855911_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/d62fb751-c5bb-4036-b9e0-ba333f7ef680_characterizeImage_855911_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a573b3b8-ad8b-4057-b434-d69f6b120e1b_isr_855911_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a573b3b8-ad8b-4057-b434-d69f6b120e1b_isr_855911_59.stderr
2024-03-09 20:55:14 proc_lsst.multi:170 [INFO]  [multi] job_id 16968976
proc_lsst.multi INFO: [multi] job_id 16968976
2024-03-09 20:55:14 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 33
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33
2024-03-09 20:55:14 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:14 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968976
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968976
parsl.executors.status_handling DEBUG: Launched block 88 on executor multi with job ID 16968976
parsl.executors.status_handling INFO: Allocated block ID 89
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-89
2024-03-09 20:55:14 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-89
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-89
2024-03-09 20:55:14 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:14 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:14 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:14 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:14 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 33
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33
2024-03-09 20:55:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:14 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:55:15 proc_lsst.multi:170 [INFO]  [multi] job_id 16968977
proc_lsst.multi INFO: [multi] job_id 16968977
2024-03-09 20:55:15 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 34
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34
2024-03-09 20:55:15 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:15 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968977
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968977
parsl.executors.status_handling DEBUG: Launched block 89 on executor multi with job ID 16968977
parsl.executors.status_handling INFO: Allocated block ID 90
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-90
2024-03-09 20:55:15 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-90
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-90
2024-03-09 20:55:15 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:15 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:15 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:15 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:15 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:15 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:15 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:15 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:15 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:15 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:15 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:15 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:15 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:15 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 34
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34
2024-03-09 20:55:15 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:15 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 235232c620a3adf251b8bce362b8ff80 with result from task 102
parsl.dataflow.dflow DEBUG: Task 102 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 36686cc28c22241ddbd9aaa70188db58
parsl.dataflow.memoization INFO: Task 103 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 407
parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/f321db45-b3e0-4ce4-a456-38b579e7c1ff_characterizeImage_855911_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/f321db45-b3e0-4ce4-a456-38b579e7c1ff_characterizeImage_855911_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/805ebeb8-9def-4e6e-8a2c-b24a2f850d6b_isr_855911_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/805ebeb8-9def-4e6e-8a2c-b24a2f850d6b_isr_855911_38.stderr
2024-03-09 20:55:15 proc_lsst.multi:170 [INFO]  [multi] job_id 16968978
proc_lsst.multi INFO: [multi] job_id 16968978
2024-03-09 20:55:15 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 35
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35
2024-03-09 20:55:15 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:15 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968978
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968978
parsl.executors.status_handling DEBUG: Launched block 90 on executor multi with job ID 16968978
parsl.executors.status_handling INFO: Allocated block ID 91
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-91
2024-03-09 20:55:15 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-91
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-91
2024-03-09 20:55:15 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:15 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:15 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:15 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:15 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:15 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:15 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:15 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:15 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:15 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:15 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:15 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:15 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:15 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 35
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35
2024-03-09 20:55:15 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:15 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 20:55:16 proc_lsst.multi:170 [INFO]  [multi] job_id 16968979
proc_lsst.multi INFO: [multi] job_id 16968979
2024-03-09 20:55:16 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 36
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36
2024-03-09 20:55:16 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:16 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968979
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968979
parsl.executors.status_handling DEBUG: Launched block 91 on executor multi with job ID 16968979
parsl.executors.status_handling INFO: Allocated block ID 92
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-92
2024-03-09 20:55:16 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-92
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-92
2024-03-09 20:55:16 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 20:55:16 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 20:55:16 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 20:55:16 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:16 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 20:55:16 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 20:55:16 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:16 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 20:55:16 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 20:55:16 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:16 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 20:55:16 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 20:55:16 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 20:55:16 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 36
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36
2024-03-09 20:55:16 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:16 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3010,198.48.92.26,169.254.95.120,10.64.129.10,10.64.65.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54457 --result_port=54826 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 25c1c892b5ac337e1e9a95a7711f5e7e with result from task 99
parsl.dataflow.dflow DEBUG: Task 99 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 100 has memoization hash ca030864dfb5ab35069f670cb4ce2c19
parsl.dataflow.memoization INFO: Task 100 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 408
parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/d5e4af58-c2aa-4647-bf1f-61b47cefb824_characterizeImage_855910_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/d5e4af58-c2aa-4647-bf1f-61b47cefb824_characterizeImage_855910_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/0321f377-6177-4332-9deb-7eb6ea92ff59_isr_855910_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/0321f377-6177-4332-9deb-7eb6ea92ff59_isr_855910_8.stderr
2024-03-09 20:55:16 proc_lsst.multi:170 [INFO]  [multi] job_id 16968980
proc_lsst.multi INFO: [multi] job_id 16968980
2024-03-09 20:55:16 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 37
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37
2024-03-09 20:55:16 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 20:55:16 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16968980
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968980
parsl.executors.status_handling DEBUG: Launched block 92 on executor multi with job ID 16968980
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 20:55:16 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 20:55:16 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6b45644cbcd85f8bdac10fb16c1c18cc with result from task 114
parsl.dataflow.dflow DEBUG: Task 114 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 115 has memoization hash afdca71199062635bf45c1e71cbafa91
parsl.dataflow.memoization INFO: Task 115 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 409
parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/cfe6de47-9909-43e3-a943-025980015432_characterizeImage_855821_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/cfe6de47-9909-43e3-a943-025980015432_characterizeImage_855821_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/37420b54-296a-412a-ad54-cebc4f021fa3_isr_855821_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/37420b54-296a-412a-ad54-cebc4f021fa3_isr_855821_32.stderr
parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 317d729119b5e6abe10c11fa34710d9f with result from task 111
parsl.dataflow.dflow DEBUG: Task 111 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 808f504107b0194da9cb323acb98a022
parsl.dataflow.memoization INFO: Task 112 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 410
parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/2779c069-44b3-4a30-a9fe-fd3b4f67ab16_characterizeImage_855909_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/2779c069-44b3-4a30-a9fe-fd3b4f67ab16_characterizeImage_855909_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/0d20626d-1345-4cc4-9752-2360c43eaadb_isr_855909_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/0d20626d-1345-4cc4-9752-2360c43eaadb_isr_855909_51.stderr
parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 04864d64154bf38c75f882b77d99ad6b with result from task 117
parsl.dataflow.dflow DEBUG: Task 117 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 1015935011ba1aec5340da4f0e1cac67
parsl.dataflow.memoization INFO: Task 118 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a40540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 411
parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/c8cbb082-36a5-4cb4-a3fe-e6634f7ebe66_characterizeImage_855822_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/c8cbb082-36a5-4cb4-a3fe-e6634f7ebe66_characterizeImage_855822_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/69a4b2ba-a675-4379-8015-19ccadf0f521_isr_855822_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/69a4b2ba-a675-4379-8015-19ccadf0f521_isr_855822_19.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:18 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7d0ca43d5bb715512d23af29cc6711ab with result from task 96
parsl.dataflow.dflow DEBUG: Task 96 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 97 has memoization hash f2f4e64e8d0b3ae7a7d5197758fe2bce
parsl.dataflow.memoization INFO: Task 97 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 412
parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/b781e9c5-cb24-41ab-abf7-9d9ed9968d05_characterizeImage_855911_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/b781e9c5-cb24-41ab-abf7-9d9ed9968d05_characterizeImage_855911_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/01558511-a086-4ce1-9e0d-bd8fe22f41de_isr_855911_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/01558511-a086-4ce1-9e0d-bd8fe22f41de_isr_855911_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:19 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:20 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:21 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:21 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:21 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:22 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:22 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:23 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:23 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:24 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 35ed5ede2ed7c917dcd7668328325ffe with result from task 159
parsl.dataflow.dflow DEBUG: Task 159 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 0fb1bd52f29ec68d0ed4b0ac9208b78a
parsl.dataflow.memoization INFO: Task 160 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a43100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 413
parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ad14a9c7-0779-40c7-9c26-b61b38e7d269_characterizeImage_855820_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ad14a9c7-0779-40c7-9c26-b61b38e7d269_characterizeImage_855820_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/f52fd248-4d46-48ad-acca-12fbda1b83cb_isr_855820_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/f52fd248-4d46-48ad-acca-12fbda1b83cb_isr_855820_1.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:24 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:25 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:25 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ad8fb20d1fc4807d6f72c3286d3e043e with result from task 153
parsl.dataflow.dflow DEBUG: Task 153 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 154 has memoization hash e46a7a47b5b5f033e660f14b23907856
parsl.dataflow.memoization INFO: Task 154 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a42ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 414
parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/7d74dfe9-174b-4646-a32d-222ccb910e73_characterizeImage_855911_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/7d74dfe9-174b-4646-a32d-222ccb910e73_characterizeImage_855911_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a0121ba7-0d79-4c5f-be04-7cbdb69d8e3a_isr_855911_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a0121ba7-0d79-4c5f-be04-7cbdb69d8e3a_isr_855911_37.stderr
parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 847bfbe2d00e2a2dfee010615fe069ff with result from task 201
parsl.dataflow.dflow DEBUG: Task 201 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 202 has memoization hash c0d66659766a9cef44d1239f370f6c1f
parsl.dataflow.memoization INFO: Task 202 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a65d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 415
parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c5a4fe4e-d413-4790-808a-ade5a22926eb_characterizeImage_855911_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c5a4fe4e-d413-4790-808a-ade5a22926eb_characterizeImage_855911_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/b3f9a143-6f60-471b-a52f-81284361f8c2_isr_855911_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/b3f9a143-6f60-471b-a52f-81284361f8c2_isr_855911_2.stderr
parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9cbd0a887d7aeed7356c073270a89ada with result from task 156
parsl.dataflow.dflow DEBUG: Task 156 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 88a6fefa3b40c60faa771672a0712bb0
parsl.dataflow.memoization INFO: Task 157 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a42de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 416
parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/52a1e22b-42aa-48c9-b101-0ef92d1dc026_characterizeImage_855909_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/52a1e22b-42aa-48c9-b101-0ef92d1dc026_characterizeImage_855909_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/79182091-7034-4fb2-a6e1-df4aabd3abb2_isr_855909_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/79182091-7034-4fb2-a6e1-df4aabd3abb2_isr_855909_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:26 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:26 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e9f486b730958364fc00a00a8a99d050 with result from task 225
parsl.dataflow.dflow DEBUG: Task 225 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 226 has memoization hash f973ad4dfaef6c726ae0baded3d4e609
parsl.dataflow.memoization INFO: Task 226 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a67600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 417
parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/eb151e5f-ad89-4460-b1d9-28231e3a34fd_characterizeImage_855821_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/eb151e5f-ad89-4460-b1d9-28231e3a34fd_characterizeImage_855821_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/6fc76c45-64ad-41c3-8f11-f5958d29a3a1_isr_855821_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/6fc76c45-64ad-41c3-8f11-f5958d29a3a1_isr_855821_33.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:27 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d155b368ee4a4dff223a963d8360991b with result from task 234
parsl.dataflow.dflow DEBUG: Task 234 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 5561f32e6f4b4b89ef1adb755cd928ff
parsl.dataflow.memoization INFO: Task 235 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a67f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 418
parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/c1f3fa98-d0bf-4f86-b700-ad0c23e1fd85_characterizeImage_855821_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/c1f3fa98-d0bf-4f86-b700-ad0c23e1fd85_characterizeImage_855821_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/3f88f97f-346f-4413-923f-af5780e90136_isr_855821_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/3f88f97f-346f-4413-923f-af5780e90136_isr_855821_31.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:27 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6ed99c68f68deb045be31f4362b180d9 with result from task 228
parsl.dataflow.dflow DEBUG: Task 228 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 229 has memoization hash 26981d5566435bb10904208869ba3e6d
parsl.dataflow.memoization INFO: Task 229 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a67920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 419
parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/64ab648d-e9a3-4190-b3f3-5d3a6cf90256_characterizeImage_855822_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/64ab648d-e9a3-4190-b3f3-5d3a6cf90256_characterizeImage_855822_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/a7d09716-7d80-47c1-b7ae-95a56c7a83a7_isr_855822_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/a7d09716-7d80-47c1-b7ae-95a56c7a83a7_isr_855822_39.stderr
parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 849d08d123ddb31f04135ed75483fd43 with result from task 216
parsl.dataflow.dflow DEBUG: Task 216 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 777406fd0ff1ff8924b850de4258a8b7
parsl.dataflow.memoization INFO: Task 217 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a66ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 420
parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/d6b371d3-6370-45e8-94dd-becb9351d7c7_characterizeImage_855820_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/d6b371d3-6370-45e8-94dd-becb9351d7c7_characterizeImage_855820_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/b6615050-2244-40e6-b26c-d50b9d9df275_isr_855820_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/b6615050-2244-40e6-b26c-d50b9d9df275_isr_855820_13.stderr
parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9b5d77f7318b1eecc51ac03bae050e7d with result from task 162
parsl.dataflow.dflow DEBUG: Task 162 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 163 has memoization hash fd9831a1355c61e6fdd07eed301095d6
parsl.dataflow.memoization INFO: Task 163 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a43420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 421
parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/3719b576-79ad-4b8a-aefe-850d3b867ec6_characterizeImage_855820_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/3719b576-79ad-4b8a-aefe-850d3b867ec6_characterizeImage_855820_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/34f02749-9576-4581-9a5f-a0bd160b8d5c_isr_855820_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/34f02749-9576-4581-9a5f-a0bd160b8d5c_isr_855820_26.stderr
parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 836c910399351fba796b5fa3b59e95ce with result from task 135
parsl.dataflow.dflow DEBUG: Task 135 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 9d66b7a2145248f52c1f9ae36e58215c
parsl.dataflow.memoization INFO: Task 136 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a41800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 422
parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/91bbf13d-1849-42e0-984a-e0e2b17d282e_characterizeImage_855910_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/91bbf13d-1849-42e0-984a-e0e2b17d282e_characterizeImage_855910_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/e03210b7-a72f-486f-9e3c-7c1a23142d69_isr_855910_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/e03210b7-a72f-486f-9e3c-7c1a23142d69_isr_855910_4.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:28 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 60b7d32becd9d99cd1fac6abc4e7ac0e with result from task 198
parsl.dataflow.dflow DEBUG: Task 198 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 199 has memoization hash a807ccd39a385f95f710aec7ff05a570
parsl.dataflow.memoization INFO: Task 199 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a659e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 423
parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/ac9d00eb-08f8-4dba-acad-e1bb1800f245_characterizeImage_855910_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/ac9d00eb-08f8-4dba-acad-e1bb1800f245_characterizeImage_855910_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/04fd684c-3ae9-4228-9cfd-4e9e7d720881_isr_855910_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/04fd684c-3ae9-4228-9cfd-4e9e7d720881_isr_855910_15.stderr
parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 77da93fb77cdb75ce3ab8909df0a3e3f with result from task 231
parsl.dataflow.dflow DEBUG: Task 231 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 18a6e3ca9ec37edc660f4ffb6b2929be
parsl.dataflow.memoization INFO: Task 232 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a67c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 424
parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/77e5bdb6-dd7e-4086-aafd-23c417e7c246_characterizeImage_855909_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/77e5bdb6-dd7e-4086-aafd-23c417e7c246_characterizeImage_855909_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/16b58f6c-d4fd-429e-a436-887720a104e7_isr_855909_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/16b58f6c-d4fd-429e-a436-887720a104e7_isr_855909_26.stderr
parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 72d8436b9bc9a2f2b9b5e15e419229af with result from task 186
parsl.dataflow.dflow DEBUG: Task 186 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 187 has memoization hash a7ee76acc3a5fd0fd879cd4fd7b422ab
parsl.dataflow.memoization INFO: Task 187 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a64d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 425
parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/380f55c6-32ea-4c0c-96bb-f30cf90e608d_characterizeImage_855911_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/380f55c6-32ea-4c0c-96bb-f30cf90e608d_characterizeImage_855911_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/d1d2a5d7-97d9-4409-9159-2470bb3f092d_isr_855911_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/d1d2a5d7-97d9-4409-9159-2470bb3f092d_isr_855911_29.stderr
parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f117d25131d3e1aed3b97b09d6946f2d with result from task 207
parsl.dataflow.dflow DEBUG: Task 207 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 77b95601efaab5aa94e501a23b281932
parsl.dataflow.memoization INFO: Task 208 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a66340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 426
parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/1772a9f8-c709-426c-ab7c-2153ab24dcfb_characterizeImage_855911_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/1772a9f8-c709-426c-ab7c-2153ab24dcfb_characterizeImage_855911_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/c978c519-2d2d-4b1e-8d44-1cb35ebfd8a9_isr_855911_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/c978c519-2d2d-4b1e-8d44-1cb35ebfd8a9_isr_855911_4.stderr
parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c1b18c0b458354117fc9318a22053b6a with result from task 180
parsl.dataflow.dflow DEBUG: Task 180 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 0b75b0a1ebdb08d2b3cc0dc4bec87a8a
parsl.dataflow.memoization INFO: Task 181 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a64720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 427
parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/702b3551-ae2c-4941-a5f3-f54f3bb80bbf_characterizeImage_855821_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/702b3551-ae2c-4941-a5f3-f54f3bb80bbf_characterizeImage_855821_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/e227bbd7-306a-41ea-9a80-8ed432204214_isr_855821_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/e227bbd7-306a-41ea-9a80-8ed432204214_isr_855821_36.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:29 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 46d4152b1850d1f77e7fce04efd72f93 with result from task 270
parsl.dataflow.dflow DEBUG: Task 270 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 271 has memoization hash 2023bebb8a8160e0d4f8e47a5fe99512
parsl.dataflow.memoization INFO: Task 271 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 428
parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/3a71f63b-37d0-4ad6-babb-85e99033a7c5_characterizeImage_855909_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/3a71f63b-37d0-4ad6-babb-85e99033a7c5_characterizeImage_855909_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/62f46889-646e-4827-9fff-99e67ee588cd_isr_855909_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/62f46889-646e-4827-9fff-99e67ee588cd_isr_855909_7.stderr
parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e9506cbe2f11da0c31611fe6bc22650a with result from task 177
parsl.dataflow.dflow DEBUG: Task 177 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 6379d4913b177dee195cdbd2309ecf9c
parsl.dataflow.memoization INFO: Task 178 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a64400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 429
parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/712ce072-1354-4115-8b52-572c14a129cd_characterizeImage_855821_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/712ce072-1354-4115-8b52-572c14a129cd_characterizeImage_855821_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/1fe0b93f-672b-4c3a-9a99-5e446c9fefc3_isr_855821_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/1fe0b93f-672b-4c3a-9a99-5e446c9fefc3_isr_855821_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:29 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c8ee658467af959038342d0c5ca30d3f with result from task 267
parsl.dataflow.dflow DEBUG: Task 267 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 268 has memoization hash b6f2a7d11ff911870720f39b20715039
parsl.dataflow.memoization INFO: Task 268 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 430
parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/dce35c21-7d97-4e16-ad95-28d61e8f3a90_characterizeImage_855820_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/dce35c21-7d97-4e16-ad95-28d61e8f3a90_characterizeImage_855820_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/f9edba97-d5fe-41d9-a1ae-d0e5dfc4b312_isr_855820_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/f9edba97-d5fe-41d9-a1ae-d0e5dfc4b312_isr_855820_55.stderr
parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1e4030f2e8dd9cebb678f2f2a30d3a3a with result from task 219
parsl.dataflow.dflow DEBUG: Task 219 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 9c25a4ce2f96c590314c552d34d94b56
parsl.dataflow.memoization INFO: Task 220 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a66fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 431
parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/bc8324a4-3014-4ca8-a5a1-34ff9e687ac7_characterizeImage_855822_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/bc8324a4-3014-4ca8-a5a1-34ff9e687ac7_characterizeImage_855822_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/70205c51-8fbb-4b29-a7b6-c7d07e53c86f_isr_855822_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/70205c51-8fbb-4b29-a7b6-c7d07e53c86f_isr_855822_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:30 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d8ac6bd5c8e0d3b22ca60a84e4945914 with result from task 237
parsl.dataflow.dflow DEBUG: Task 237 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 2929d5ef1b94dfc00498f4de6509140c
parsl.dataflow.memoization INFO: Task 238 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 432
parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/8b099f98-e72e-49c4-a069-956186a7ca4b_characterizeImage_855820_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/8b099f98-e72e-49c4-a069-956186a7ca4b_characterizeImage_855820_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/ac1ee20a-9d1c-4e75-bfd8-853a271fbe0d_isr_855820_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/ac1ee20a-9d1c-4e75-bfd8-853a271fbe0d_isr_855820_2.stderr
parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 37acff8635e4a7bed0a76de2cefbdc68 with result from task 222
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:30 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow DEBUG: Task 222 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 9c84d7ca4a951680114701fdadacc267
parsl.dataflow.memoization INFO: Task 223 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a672e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 433
parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/e17ff768-72f8-4967-91a7-905d931ecf88_characterizeImage_855821_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/e17ff768-72f8-4967-91a7-905d931ecf88_characterizeImage_855821_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/7d67734a-124e-44eb-bb94-dee89d6d1a5d_isr_855821_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/7d67734a-124e-44eb-bb94-dee89d6d1a5d_isr_855821_37.stderr
parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f83d3ee443c5fdd59a29293fb17c4b5b with result from task 174
parsl.dataflow.dflow DEBUG: Task 174 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 7b44b8d8d954dd25b90ce1c4ba93fe22
parsl.dataflow.memoization INFO: Task 175 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a43ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 434
parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/f50baa40-956a-4cea-9495-2f8bb14c5c52_characterizeImage_855822_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/f50baa40-956a-4cea-9495-2f8bb14c5c52_characterizeImage_855822_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/3f5c447f-f558-4219-82ed-56cc3791e0bf_isr_855822_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/3f5c447f-f558-4219-82ed-56cc3791e0bf_isr_855822_57.stderr
parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9e6131faa9da0dc00b068350bfbcd4c7 with result from task 240
parsl.dataflow.dflow DEBUG: Task 240 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 5fb14ac0b1e2308b3793fda63da97574
parsl.dataflow.memoization INFO: Task 241 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 435
parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/c7c3ea5e-8a0c-4877-a737-7010f274c97a_characterizeImage_855822_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/c7c3ea5e-8a0c-4877-a737-7010f274c97a_characterizeImage_855822_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/8414eacf-78a0-4725-8534-e2d8ff9d78a1_isr_855822_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/8414eacf-78a0-4725-8534-e2d8ff9d78a1_isr_855822_20.stderr
parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 65b8a0378a15bcbf4aa56cc0447a67fd with result from task 210
parsl.dataflow.dflow DEBUG: Task 210 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 211 has memoization hash 2c663416d7adb2abfdd56c12c4f24d02
parsl.dataflow.memoization INFO: Task 211 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a66660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 436
parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/4dc55871-ed7d-4f58-a64a-27716c62a45e_characterizeImage_855909_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/4dc55871-ed7d-4f58-a64a-27716c62a45e_characterizeImage_855909_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/298f3187-2d40-4eef-8c3f-5510bfe7624b_isr_855909_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/298f3187-2d40-4eef-8c3f-5510bfe7624b_isr_855909_9.stderr
parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bc03486a6d82701207ed5d69cd79a2b9 with result from task 276
parsl.dataflow.dflow DEBUG: Task 276 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 277 has memoization hash d33ef21e16fc9ade18cc01e249ed93c5
parsl.dataflow.memoization INFO: Task 277 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 437
parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/7fcf69f2-8a37-47ff-bd9d-99e0d12220bb_characterizeImage_855911_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/7fcf69f2-8a37-47ff-bd9d-99e0d12220bb_characterizeImage_855911_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a924cadd-4789-48e5-bf78-7a16efcc32d5_isr_855911_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a924cadd-4789-48e5-bf78-7a16efcc32d5_isr_855911_34.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:31 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6f42ea689036f94223bd9d7797de077e with result from task 189
parsl.dataflow.dflow DEBUG: Task 189 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 190 has memoization hash c72d3eb61df749b74d5376420f48a018
parsl.dataflow.memoization INFO: Task 190 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a65080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 438
parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/1c2b6108-4f77-4ad5-8b16-eb1d8b97c7d5_characterizeImage_855820_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/1c2b6108-4f77-4ad5-8b16-eb1d8b97c7d5_characterizeImage_855820_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/900274fa-504c-45ea-b55a-3c751c69723a_isr_855820_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/900274fa-504c-45ea-b55a-3c751c69723a_isr_855820_29.stderr
parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bb13296f9ac2ab57568f0cd0883a9e94 with result from task 285
parsl.dataflow.dflow DEBUG: Task 285 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 286 has memoization hash d44d1ad2dc0741a55b0dadf23230749e
parsl.dataflow.memoization INFO: Task 286 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 439
parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/154738c4-bdef-4810-87ee-747edb340511_characterizeImage_855822_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/154738c4-bdef-4810-87ee-747edb340511_characterizeImage_855822_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/a80963c7-fa46-4ca0-b8f5-055c94f4bd09_isr_855822_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/a80963c7-fa46-4ca0-b8f5-055c94f4bd09_isr_855822_46.stderr
parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8f3aed92e4af93734e6efbc7f8ddedbd with result from task 171
parsl.dataflow.dflow DEBUG: Task 171 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 2b64f44f87a74058a58e3de9a0a7b218
parsl.dataflow.memoization INFO: Task 172 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a43d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 440
parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/49cb6f63-543d-4e57-a501-5cd8739737fa_characterizeImage_855820_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/49cb6f63-543d-4e57-a501-5cd8739737fa_characterizeImage_855820_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/e554d0ba-a1e1-44e6-a4e1-193ec1cb3a23_isr_855820_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/e554d0ba-a1e1-44e6-a4e1-193ec1cb3a23_isr_855820_37.stderr
parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2f5251661c60eb6a330982c924b3d2ec with result from task 195
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:31 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow DEBUG: Task 195 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 196 has memoization hash 18421befe5b73c0ed1a797a04499c87f
parsl.dataflow.memoization INFO: Task 196 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a656c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 441
parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/2ef67a39-112c-4dad-b8df-1174db3d4512_characterizeImage_855822_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/2ef67a39-112c-4dad-b8df-1174db3d4512_characterizeImage_855822_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/e45fa78f-748c-455a-b7fd-a46298f0f6ae_isr_855822_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/e45fa78f-748c-455a-b7fd-a46298f0f6ae_isr_855822_59.stderr
parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2d17026c4e91a60069ddf3d31192869d with result from task 255
parsl.dataflow.dflow DEBUG: Task 255 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 256 has memoization hash 2d7a2101134f7571e7b2e7ff90f73aca
parsl.dataflow.memoization INFO: Task 256 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 442
parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/3f7cd907-ec3f-4f1e-b09e-390a4ad9fa74_characterizeImage_855820_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/3f7cd907-ec3f-4f1e-b09e-390a4ad9fa74_characterizeImage_855820_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/afe62168-58bf-461a-a93a-910a2b626907_isr_855820_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/afe62168-58bf-461a-a93a-910a2b626907_isr_855820_33.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:32 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 788b9532b301cfe477c203eebe8b951c with result from task 204
parsl.dataflow.dflow DEBUG: Task 204 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 205 has memoization hash c4da951aec1cde9c135aa677eacde2e7
parsl.dataflow.memoization INFO: Task 205 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a66020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 443
parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a5fefbfe-5e7b-4e1a-976e-347e18a8306b_characterizeImage_855910_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a5fefbfe-5e7b-4e1a-976e-347e18a8306b_characterizeImage_855910_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/b83c54be-4ac7-4baa-b4b8-39fcb809ac4a_isr_855910_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/b83c54be-4ac7-4baa-b4b8-39fcb809ac4a_isr_855910_49.stderr
parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 065be44d650c33b2de381b55514efe30 with result from task 261
parsl.dataflow.dflow DEBUG: Task 261 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 262 has memoization hash 1c07456e656a1da98ecd38606437dc9f
parsl.dataflow.memoization INFO: Task 262 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 444
parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/4775d0e9-750f-4afd-977d-eac5c6e7559f_characterizeImage_855820_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/4775d0e9-750f-4afd-977d-eac5c6e7559f_characterizeImage_855820_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0686c199-db34-403f-aec5-3d7290adf814_isr_855820_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0686c199-db34-403f-aec5-3d7290adf814_isr_855820_25.stderr
parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 438f62b25463505aeaefd6df4561fa0d with result from task 249
parsl.dataflow.dflow DEBUG: Task 249 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 250 has memoization hash 77a2de79be55f7c38e2c4686779bfc40
parsl.dataflow.memoization INFO: Task 250 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 445
parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/dadb0dfe-a789-4d3d-87dc-8734e7364f25_characterizeImage_855822_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/dadb0dfe-a789-4d3d-87dc-8734e7364f25_characterizeImage_855822_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/f5601c98-96f5-4974-9611-d091f863f617_isr_855822_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/f5601c98-96f5-4974-9611-d091f863f617_isr_855822_15.stderr
parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 004628d512f427f5c536f061e1283c9a with result from task 165
parsl.dataflow.dflow DEBUG: Task 165 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 166 has memoization hash e904e688924c40b988c4473675d4c75f
parsl.dataflow.memoization INFO: Task 166 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a43740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 446
parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/d60acd2b-0de2-4151-8a8c-3093be5bba4c_characterizeImage_855822_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/d60acd2b-0de2-4151-8a8c-3093be5bba4c_characterizeImage_855822_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/3f667c69-86fb-4dfb-a8b5-323d038fa19e_isr_855822_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/3f667c69-86fb-4dfb-a8b5-323d038fa19e_isr_855822_25.stderr
parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 307b9bb7d532732c39cf51bf05795553 with result from task 264
parsl.dataflow.dflow DEBUG: Task 264 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 265 has memoization hash a3ef10131e9972f6fafacc9d1ba6b070
parsl.dataflow.memoization INFO: Task 265 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 447
parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/69ea0eae-924f-482c-81db-93e497d5146f_characterizeImage_855909_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/69ea0eae-924f-482c-81db-93e497d5146f_characterizeImage_855909_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/a38e9d92-e7a9-4662-94c0-502d800146d9_isr_855909_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/a38e9d92-e7a9-4662-94c0-502d800146d9_isr_855909_22.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:32 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6def333686093877731d4fd9c53eb41c with result from task 213
parsl.dataflow.dflow DEBUG: Task 213 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 214 has memoization hash ea5d22a6f4dbcf365da3493aa1f54f04
parsl.dataflow.memoization INFO: Task 214 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a66980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 448
parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/0dc84a3a-9b16-4880-86a7-cbd6b07b9bf0_characterizeImage_855820_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/0dc84a3a-9b16-4880-86a7-cbd6b07b9bf0_characterizeImage_855820_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/08242780-7300-46c2-b518-ea60416ae3ba_isr_855820_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/08242780-7300-46c2-b518-ea60416ae3ba_isr_855820_18.stderr
parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 174e89e6d237eaecb54e42c2f9d1766e with result from task 183
parsl.dataflow.dflow DEBUG: Task 183 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 184 has memoization hash a93c3a47d61ca93ffb05f3875603c976
parsl.dataflow.memoization INFO: Task 184 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a64a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 449
parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/bb148c1a-6ace-4fd1-a3ae-0a96242c60c5_characterizeImage_855821_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/bb148c1a-6ace-4fd1-a3ae-0a96242c60c5_characterizeImage_855821_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/c49b272e-0b97-499d-bd6a-99abd921e485_isr_855821_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/c49b272e-0b97-499d-bd6a-99abd921e485_isr_855821_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:33 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:33 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e4cd8e62774ad5c12e9005926c3b7869 with result from task 246
parsl.dataflow.dflow DEBUG: Task 246 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 247 has memoization hash c6256a479ac4a6dce42410bf92f34d4c
parsl.dataflow.memoization INFO: Task 247 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 450
parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/37cf5ddc-1512-4df8-8352-0b285e7f8602_characterizeImage_855909_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/37cf5ddc-1512-4df8-8352-0b285e7f8602_characterizeImage_855909_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/40e896a3-b962-4cef-b188-e209f28ee26b_isr_855909_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/40e896a3-b962-4cef-b188-e209f28ee26b_isr_855909_29.stderr
parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 34ea81b45f19c96efcf2a45dadcbf72f with result from task 192
parsl.dataflow.dflow DEBUG: Task 192 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 193 has memoization hash 7b84f213a8135c1c93935a5ec379c857
parsl.dataflow.memoization INFO: Task 193 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a653a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 451
parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/83efddc5-f05c-4a6b-8c39-408e482bcd01_characterizeImage_855822_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/83efddc5-f05c-4a6b-8c39-408e482bcd01_characterizeImage_855822_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/4d6a5155-dcff-4461-b6ff-044929cbae98_isr_855822_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/4d6a5155-dcff-4461-b6ff-044929cbae98_isr_855822_60.stderr
parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3ea4c1143889660badcdd28a4d41a834 with result from task 123
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:34 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.dataflow.dflow DEBUG: Task 123 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 124 has memoization hash b2a814da0e5557dbab2e9f18196977b6
parsl.dataflow.memoization INFO: Task 124 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a40b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 452
parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/e738178b-8a53-4006-a114-92a65e61b365_characterizeImage_855822_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/e738178b-8a53-4006-a114-92a65e61b365_characterizeImage_855822_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/2aedf1d3-7761-4c55-8ce7-cc04a950b656_isr_855822_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/2aedf1d3-7761-4c55-8ce7-cc04a950b656_isr_855822_56.stderr
parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0a5eeb2b5c80909f7a5f23f667a4e0bd with result from task 120
parsl.dataflow.dflow DEBUG: Task 120 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 3e512ad8a52b83510ec1ab4f2a8bd94c
parsl.dataflow.memoization INFO: Task 121 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a40860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 453
parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/da5eed5a-7ad1-4ef6-9cc0-dd0c21c150bb_characterizeImage_855820_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/da5eed5a-7ad1-4ef6-9cc0-dd0c21c150bb_characterizeImage_855820_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/8a779896-41bc-4ef4-9a47-67ad503b3943_isr_855820_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/8a779896-41bc-4ef4-9a47-67ad503b3943_isr_855820_24.stderr
parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3273fb93c26ce1c3d394e02db55464a6 with result from task 150
parsl.dataflow.dflow DEBUG: Task 150 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 339ad9c3326aedaa127a2800a4103eb2
parsl.dataflow.memoization INFO: Task 151 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a427a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 454
parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/96a3abcb-659e-4f2d-94ac-7e1b4c5d3f86_characterizeImage_855911_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/96a3abcb-659e-4f2d-94ac-7e1b4c5d3f86_characterizeImage_855911_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/749780eb-505f-4244-a361-18678705070f_isr_855911_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/749780eb-505f-4244-a361-18678705070f_isr_855911_41.stderr
parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d7f44461bc47f85d3206462644403b09 with result from task 141
parsl.dataflow.dflow DEBUG: Task 141 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 4cbb3e3e9ab9f5306f6f725e3e892615
parsl.dataflow.memoization INFO: Task 142 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a41e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 455
parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/db6bc5f1-fc1e-4c4f-9dc8-b2f21acf45f6_characterizeImage_855821_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/db6bc5f1-fc1e-4c4f-9dc8-b2f21acf45f6_characterizeImage_855821_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/832fd814-a0e1-4a16-b7cc-3e83e1086855_isr_855821_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/832fd814-a0e1-4a16-b7cc-3e83e1086855_isr_855821_30.stderr
parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f56780f70044465f6f76116c3da76edb with result from task 147
parsl.dataflow.dflow DEBUG: Task 147 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 90a5e7a6e4fa6c5c21698fdc2261309f
parsl.dataflow.memoization INFO: Task 148 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a42480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 456
parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/82b6904a-d982-4b87-ae72-5032512b470b_characterizeImage_855822_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/82b6904a-d982-4b87-ae72-5032512b470b_characterizeImage_855822_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/48ef93b4-3555-465c-84f3-9b57e3cc6e4b_isr_855822_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/48ef93b4-3555-465c-84f3-9b57e3cc6e4b_isr_855822_23.stderr
parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bbd2cd797e781871890c20d5fa504752 with result from task 144
parsl.dataflow.dflow DEBUG: Task 144 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 62f3d0459baa14ae7b6674ecf5ecc238
parsl.dataflow.memoization INFO: Task 145 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a42160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 457
parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/2596cad3-9b26-4917-bb13-8b4035f77480_characterizeImage_855910_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/2596cad3-9b26-4917-bb13-8b4035f77480_characterizeImage_855910_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/7603e76f-23e0-4c0d-ad21-920bc8b270a3_isr_855910_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/7603e76f-23e0-4c0d-ad21-920bc8b270a3_isr_855910_34.stderr
parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5156c9c563c478edee5410b508e50f7e with result from task 138
parsl.dataflow.dflow DEBUG: Task 138 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 3accece77c75787f04ac55135ff1f658
parsl.dataflow.memoization INFO: Task 139 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a41b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 458
parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/383918ae-d7ff-48a6-8b43-40d3b29fd103_characterizeImage_855820_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/383918ae-d7ff-48a6-8b43-40d3b29fd103_characterizeImage_855820_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/fd3492ab-a3a9-4a2d-95b5-7b2be7672773_isr_855820_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/fd3492ab-a3a9-4a2d-95b5-7b2be7672773_isr_855820_47.stderr
parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 975119d87b02695f31e20febae116d0d with result from task 132
parsl.dataflow.dflow DEBUG: Task 132 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 133 has memoization hash ce667f0850ca86143dd441847af193fa
parsl.dataflow.memoization INFO: Task 133 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a414e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 459
parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/0d47caae-2778-4361-af5c-f933cacb6268_characterizeImage_855821_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/0d47caae-2778-4361-af5c-f933cacb6268_characterizeImage_855821_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/b6adf367-2025-4337-8221-49d15cdbb732_isr_855821_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/b6adf367-2025-4337-8221-49d15cdbb732_isr_855821_34.stderr
parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 95d718212725e58b55316f495456c8ba with result from task 273
parsl.dataflow.dflow DEBUG: Task 273 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 85cb0c51440dcd0df1aa0359cc43cde0
parsl.dataflow.memoization INFO: Task 274 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 460
parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/6df5a330-f7f2-449a-9a11-a744dffbcd6a_characterizeImage_855910_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/6df5a330-f7f2-449a-9a11-a744dffbcd6a_characterizeImage_855910_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/8b0c070e-1b36-46b5-9c12-2434db50fc58_isr_855910_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/8b0c070e-1b36-46b5-9c12-2434db50fc58_isr_855910_52.stderr
parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8a95dba507e9084065923a36eb6f040b with result from task 252
parsl.dataflow.dflow DEBUG: Task 252 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 253 has memoization hash b28f369b3faea4b21e8e1be6c45c0cd2
parsl.dataflow.memoization INFO: Task 253 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 461
parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9b4ba870-7e1b-465c-a0ab-7fa4bb4af0c8_characterizeImage_855910_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9b4ba870-7e1b-465c-a0ab-7fa4bb4af0c8_characterizeImage_855910_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/c8aacc6c-64c7-4f89-a359-1df47c18d57d_isr_855910_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/c8aacc6c-64c7-4f89-a359-1df47c18d57d_isr_855910_7.stderr
parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 761d7c527345e72464d2a030660537ac with result from task 258
parsl.dataflow.dflow DEBUG: Task 258 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 259 has memoization hash a14b9c0bcfa4a65c9d8badbf0b357b4e
parsl.dataflow.memoization INFO: Task 259 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 462
parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/11397273-b5f0-4be1-a654-807721f28642_characterizeImage_855821_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/11397273-b5f0-4be1-a654-807721f28642_characterizeImage_855821_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/9f572600-2953-4721-9d09-60c594a9f0d5_isr_855821_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/9f572600-2953-4721-9d09-60c594a9f0d5_isr_855821_38.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:34 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:35 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c3397154156a59932666e7e61a96105f with result from task 126
parsl.dataflow.dflow DEBUG: Task 126 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 7466995a82c743d617d8aac6eb0df392
parsl.dataflow.memoization INFO: Task 127 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a40ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 463
parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/7d0588e2-8f78-4848-8899-7486d9b8b2b3_characterizeImage_855909_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/7d0588e2-8f78-4848-8899-7486d9b8b2b3_characterizeImage_855909_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/9232864f-0d9b-4a55-9656-45cf0de8ce48_isr_855909_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/9232864f-0d9b-4a55-9656-45cf0de8ce48_isr_855909_60.stderr
parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f35ecd19dcb1687552952d854241c293 with result from task 129
parsl.dataflow.dflow DEBUG: Task 129 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 6eecabf2fbcccec0bc5b18934fad458d
parsl.dataflow.memoization INFO: Task 130 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a411c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 464
parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/babd6655-7ce5-4492-8167-f25fe8eedc2f_characterizeImage_855909_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/babd6655-7ce5-4492-8167-f25fe8eedc2f_characterizeImage_855909_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/97b22ee0-7f3d-418b-b816-1e4727cc2497_isr_855909_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/97b22ee0-7f3d-418b-b816-1e4727cc2497_isr_855909_59.stderr
parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 87a74bcf96efbff4c5aef5d4099f71b9 with result from task 243
parsl.dataflow.dflow DEBUG: Task 243 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 5b152b1eeb38cdbcff24e57446f75e3d
parsl.dataflow.memoization INFO: Task 244 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 465
parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/41534af1-32a6-43bf-b993-c06516b8dd3c_characterizeImage_855820_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/41534af1-32a6-43bf-b993-c06516b8dd3c_characterizeImage_855820_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/4fdfc300-9543-4666-9c9d-75ee1ed00eaa_isr_855820_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/4fdfc300-9543-4666-9c9d-75ee1ed00eaa_isr_855820_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:35 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:36 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:36 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ec180dc5c35eadb124205678a7d28fa8 with result from task 168
parsl.dataflow.dflow DEBUG: Task 168 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 169 has memoization hash b6b49f24a76c9d7e90f3eb9dca21751d
parsl.dataflow.memoization INFO: Task 169 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a43a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 466
parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/193d03cc-4830-466f-833e-b737dadc3534_characterizeImage_855909_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/193d03cc-4830-466f-833e-b737dadc3534_characterizeImage_855909_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/fff4024b-5906-4bb6-8081-44839be3539b_isr_855909_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/fff4024b-5906-4bb6-8081-44839be3539b_isr_855909_43.stderr
parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 587d74c0296596eb17e4530d10b1f74d with result from task 282
parsl.dataflow.dflow DEBUG: Task 282 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 283 has memoization hash d5cf1e2b92bd1f8c58cc6f70de9283b5
parsl.dataflow.memoization INFO: Task 283 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 467
parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/f88194eb-8cbd-4bcd-877a-edaec2c0ce55_characterizeImage_855910_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/f88194eb-8cbd-4bcd-877a-edaec2c0ce55_characterizeImage_855910_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/9491ef52-c0bb-4376-a38f-825a617ac659_isr_855910_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/9491ef52-c0bb-4376-a38f-825a617ac659_isr_855910_41.stderr
parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 143790abd175443e706e0c02e130e6f6 with result from task 279
parsl.dataflow.dflow DEBUG: Task 279 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 5d02436569fab31be142ce7d09f174e9
parsl.dataflow.memoization INFO: Task 280 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 468
parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/4347a38c-3c1a-4b9d-b7a5-03d8abc5003b_characterizeImage_855821_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/4347a38c-3c1a-4b9d-b7a5-03d8abc5003b_characterizeImage_855821_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/35ba0be6-1152-488c-93bf-a42ee0243dbe_isr_855821_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/35ba0be6-1152-488c-93bf-a42ee0243dbe_isr_855821_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:37 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:37 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:37 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:38 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:38 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:39 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:39 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:40 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:40 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 510 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c3ff9f06f8beb6c0adccbb8d9db16a1d with result from task 510
parsl.dataflow.dflow DEBUG: Task 510 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 511 has memoization hash c8f242c7b9e33d0220c71832797d4823
parsl.dataflow.memoization INFO: Task 511 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15946020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 511 try 0 launched on executor multi with executor id 469
parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/0fa21b17-4bee-4749-be6f-82640d5b659f_characterizeImage_855911_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/0fa21b17-4bee-4749-be6f-82640d5b659f_characterizeImage_855911_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/e4fb2090-5b57-4d89-865b-d731d42bef35_isr_855911_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/e4fb2090-5b57-4d89-865b-d731d42bef35_isr_855911_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:40 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:40 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 395f7cc10aba936c04a0f398542ef1c3 with result from task 312
parsl.dataflow.dflow DEBUG: Task 312 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 313 has memoization hash a73354e138dcf9639491de69a740b98c
parsl.dataflow.memoization INFO: Task 313 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 470
parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8f7cdbb3-2180-4a76-a27b-d6bf52a1eebd_characterizeImage_855911_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8f7cdbb3-2180-4a76-a27b-d6bf52a1eebd_characterizeImage_855911_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/f34a1d8c-66cd-43ed-a810-1dc7d5ac4ab1_isr_855911_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/f34a1d8c-66cd-43ed-a810-1dc7d5ac4ab1_isr_855911_15.stderr
parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 18adfbe3bcc40402b50b1c34864bb9e0 with result from task 294
parsl.dataflow.dflow DEBUG: Task 294 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 295 has memoization hash 41d70da2e315fa561af6a6a6f51b6e77
parsl.dataflow.memoization INFO: Task 295 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 471
parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/0dd15248-135f-4cfb-8767-9155a894800d_characterizeImage_855910_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/0dd15248-135f-4cfb-8767-9155a894800d_characterizeImage_855910_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/ddc9536e-a6c5-4f78-8f7c-4e28a9238e52_isr_855910_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/ddc9536e-a6c5-4f78-8f7c-4e28a9238e52_isr_855910_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:41 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 372 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f67634ad5e45bf1e9dfae4927008a446 with result from task 372
parsl.dataflow.dflow DEBUG: Task 372 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 373 has memoization hash cb69a8e81091c970c23065e7690154af
parsl.dataflow.memoization INFO: Task 373 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 472
parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/5f20bc73-308d-49d7-b631-f3754c89a722_characterizeImage_855910_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/5f20bc73-308d-49d7-b631-f3754c89a722_characterizeImage_855910_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/fd4b610e-c965-4d35-956f-35b2b9ac3102_isr_855910_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/fd4b610e-c965-4d35-956f-35b2b9ac3102_isr_855910_32.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:41 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9ec442ed9293ad106e7c054966754659 with result from task 348
parsl.dataflow.dflow DEBUG: Task 348 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 349 has memoization hash f83a1aaa7a7d9e6e21a2bb943c52e1b1
parsl.dataflow.memoization INFO: Task 349 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16abb6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 473
parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/9527a6ba-e7ee-4870-9eaf-19618fcef374_characterizeImage_855909_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/9527a6ba-e7ee-4870-9eaf-19618fcef374_characterizeImage_855909_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/8a7ab59f-dcea-408e-8797-a9e5a0053e21_isr_855909_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/8a7ab59f-dcea-408e-8797-a9e5a0053e21_isr_855909_25.stderr
parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 79ecf59551fb9d9065a5bbca7377e850 with result from task 300
parsl.dataflow.dflow DEBUG: Task 300 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 301 has memoization hash ff2d1a10240978946e6b1ec406e8d745
parsl.dataflow.memoization INFO: Task 301 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 474
parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/a8250aee-2c03-4096-ac18-2a3ba4949160_characterizeImage_855821_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/a8250aee-2c03-4096-ac18-2a3ba4949160_characterizeImage_855821_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/fcd2c3f3-7e80-45c5-865f-a05ecf7173e3_isr_855821_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/fcd2c3f3-7e80-45c5-865f-a05ecf7173e3_isr_855821_15.stderr
parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 867b29ec6a5200ecd8a249478f938a26 with result from task 297
parsl.dataflow.dflow DEBUG: Task 297 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 298 has memoization hash 346eb0a4d3474b48938c7a1eadffd1be
parsl.dataflow.memoization INFO: Task 298 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 475
parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ef28d0f7-f8a0-4f1d-bdb4-6d1dbbf188ad_characterizeImage_855821_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ef28d0f7-f8a0-4f1d-bdb4-6d1dbbf188ad_characterizeImage_855821_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/4ef3fe48-deed-4335-99ea-11054d9f188d_isr_855821_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/4ef3fe48-deed-4335-99ea-11054d9f188d_isr_855821_18.stderr
parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 109af9534f35257785143ab312547839 with result from task 369
parsl.dataflow.dflow DEBUG: Task 369 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 370 has memoization hash c01a7dbf4f56fa06026a82b866dccf1d
parsl.dataflow.memoization INFO: Task 370 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 476
parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/17d4071f-0f92-4732-8a33-e6d45e1b118d_characterizeImage_855911_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/17d4071f-0f92-4732-8a33-e6d45e1b118d_characterizeImage_855911_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/94573edc-3979-4900-88f5-18ade0306ab8_isr_855911_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/94573edc-3979-4900-88f5-18ade0306ab8_isr_855911_19.stderr
parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fde7126bf517bad362e6ffaf50eb5830 with result from task 321
parsl.dataflow.dflow DEBUG: Task 321 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 322 has memoization hash 1aaedeabe0145fe666808f4d0d5a0ce5
parsl.dataflow.memoization INFO: Task 322 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 477
parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/c992bb54-85b1-4bcd-bdc5-5ed286804fdf_characterizeImage_855909_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/c992bb54-85b1-4bcd-bdc5-5ed286804fdf_characterizeImage_855909_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/7e5a8b03-65e3-4c90-9e1e-2c59fcd23dc5_isr_855909_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/7e5a8b03-65e3-4c90-9e1e-2c59fcd23dc5_isr_855909_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:42 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b7bff0fb81f10c26163f2a07aec70fed with result from task 357
parsl.dataflow.dflow DEBUG: Task 357 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 358 has memoization hash 58020a0853648c5d6ad8e007758a8af2
parsl.dataflow.memoization INFO: Task 358 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16abbe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 478
parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/94cc707b-1eba-45e3-ac0f-986648eb3d85_characterizeImage_855909_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/94cc707b-1eba-45e3-ac0f-986648eb3d85_characterizeImage_855909_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/222b63fe-32d7-489a-887f-648c77f23dea_isr_855909_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/222b63fe-32d7-489a-887f-648c77f23dea_isr_855909_24.stderr
parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5b3b0429f94a1fdbdc7fe7bd35c8be77 with result from task 330
parsl.dataflow.dflow DEBUG: Task 330 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 31c98fd7bcb29d807bc43f5178c741c7
parsl.dataflow.memoization INFO: Task 331 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16aba3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 479
parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/43ac2e76-0ffa-420b-952a-048e40aa8647_characterizeImage_855911_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/43ac2e76-0ffa-420b-952a-048e40aa8647_characterizeImage_855911_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/b5b71f3b-85ef-4711-bc95-53a25f91d9cc_isr_855911_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/b5b71f3b-85ef-4711-bc95-53a25f91d9cc_isr_855911_23.stderr
parsl.dataflow.dflow INFO: Task 384 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9fe2cf8493893579e9e1d260dda8bc3c with result from task 384
parsl.dataflow.dflow DEBUG: Task 384 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 385 has memoization hash 03c2727bacce02c71886883c1d3323ab
parsl.dataflow.memoization INFO: Task 385 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 480
parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/51d5a040-8a5d-4d8b-a9ec-8cd9c89fdc68_characterizeImage_855821_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/51d5a040-8a5d-4d8b-a9ec-8cd9c89fdc68_characterizeImage_855821_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/a6f434d5-1d98-4472-92fa-1dba08960f40_isr_855821_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/a6f434d5-1d98-4472-92fa-1dba08960f40_isr_855821_51.stderr
parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 97dea8dd8307977ea171a6f5c2c5f06b with result from task 303
parsl.dataflow.dflow DEBUG: Task 303 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 304 has memoization hash 57547085f1eeb75b96682f005ae8d80a
parsl.dataflow.memoization INFO: Task 304 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 481
parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/80c1e7a7-676b-4941-9117-406eb14c20cf_characterizeImage_855909_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/80c1e7a7-676b-4941-9117-406eb14c20cf_characterizeImage_855909_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/095f7b42-f9c2-4c21-9eef-2c4822825944_isr_855909_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/095f7b42-f9c2-4c21-9eef-2c4822825944_isr_855909_5.stderr
parsl.dataflow.dflow INFO: Task 378 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 81a0dd1f4a3c31ff9c3aa71c28ad084c with result from task 378
parsl.dataflow.dflow DEBUG: Task 378 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 379 has memoization hash e14e5fa0c5b18b79c04e7d599602fdec
parsl.dataflow.memoization INFO: Task 379 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 482
parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/6ac9e124-f654-4a89-882a-46f793ae9dd8_characterizeImage_855821_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/6ac9e124-f654-4a89-882a-46f793ae9dd8_characterizeImage_855821_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/3b2b70dc-8090-43ed-8ebb-7c2b4c5c737f_isr_855821_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/3b2b70dc-8090-43ed-8ebb-7c2b4c5c737f_isr_855821_48.stderr
parsl.dataflow.dflow INFO: Task 375 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 896cce030c3b70c2e4cf2d95a8f98844 with result from task 375
parsl.dataflow.dflow DEBUG: Task 375 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 376 has memoization hash 62ed2c34fff1958fae1bd9a3bb74d2f3
parsl.dataflow.memoization INFO: Task 376 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 483
parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/b261b79e-ae50-40da-a9a2-759838909c11_characterizeImage_855822_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/b261b79e-ae50-40da-a9a2-759838909c11_characterizeImage_855822_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/9e6b6fdb-6aa1-4b7f-8036-5c6e0153dece_isr_855822_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/9e6b6fdb-6aa1-4b7f-8036-5c6e0153dece_isr_855822_42.stderr
parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 55f8a2ef4ad651507a0b8d67d2202c8b with result from task 336
parsl.dataflow.dflow DEBUG: Task 336 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 337 has memoization hash 60aba6e9537b6d4dedd336405005ba02
parsl.dataflow.memoization INFO: Task 337 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16abaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 484
parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/4c5337f9-6458-446a-bce2-ca968e7d705b_characterizeImage_855820_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/4c5337f9-6458-446a-bce2-ca968e7d705b_characterizeImage_855820_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/415346ad-35b0-43cb-ac63-e1058f7ecc2f_isr_855820_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/415346ad-35b0-43cb-ac63-e1058f7ecc2f_isr_855820_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:42 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 37fea1429df7cbc9028861091e68b01f with result from task 333
parsl.dataflow.dflow DEBUG: Task 333 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 334 has memoization hash b3f81b15c72d5032dbbb89d0dcb5994a
parsl.dataflow.memoization INFO: Task 334 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16aba700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 485
parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/9e1bac5a-f371-45c8-8550-39c325de8911_characterizeImage_855822_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/9e1bac5a-f371-45c8-8550-39c325de8911_characterizeImage_855822_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/0193b4c6-7d2f-4e46-bf89-c0ef4beaf065_isr_855822_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/0193b4c6-7d2f-4e46-bf89-c0ef4beaf065_isr_855822_62.stderr
parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6981e93ead115aa73a53749d111525ef with result from task 306
parsl.dataflow.dflow DEBUG: Task 306 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 307 has memoization hash b638ea1d813b2bd661d044d5f67a3d62
parsl.dataflow.memoization INFO: Task 307 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 486
parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9ed1bf3e-ea29-498b-8e9b-247ecfad18e0_characterizeImage_855910_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9ed1bf3e-ea29-498b-8e9b-247ecfad18e0_characterizeImage_855910_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/4e613f7e-ddcc-4fc4-8ce9-8a7ab65b5c1e_isr_855910_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/4e613f7e-ddcc-4fc4-8ce9-8a7ab65b5c1e_isr_855910_47.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:43 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dbea3445af9489fa97b12f9c027d34de with result from task 354
parsl.dataflow.dflow DEBUG: Task 354 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 355 has memoization hash 8c591d801a49ded05e4053bf75bb6bff
parsl.dataflow.memoization INFO: Task 355 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16abbce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 487
parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/ab0f61a5-3ad2-4dec-8705-232145f6b45e_characterizeImage_855911_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/ab0f61a5-3ad2-4dec-8705-232145f6b45e_characterizeImage_855911_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/540b7053-14eb-4abf-8593-64f71ad7acb0_isr_855911_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/540b7053-14eb-4abf-8593-64f71ad7acb0_isr_855911_1.stderr
parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ae30bc58307758fc2a125122643d20ea with result from task 342
parsl.dataflow.dflow DEBUG: Task 342 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 343 has memoization hash 992d02b07d06394b7645335d1e50fe96
parsl.dataflow.memoization INFO: Task 343 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16abb060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 488
parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/1bcf117f-bad4-4fc6-8fd0-b2b31835728d_characterizeImage_855910_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/1bcf117f-bad4-4fc6-8fd0-b2b31835728d_characterizeImage_855910_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/5c803b1f-c8d7-4978-97e4-104d602c4ed1_isr_855910_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/5c803b1f-c8d7-4978-97e4-104d602c4ed1_isr_855910_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:43 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 23ef97170ab3d05d87d8f717047f0955 with result from task 309
parsl.dataflow.dflow DEBUG: Task 309 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 116741d08cc736010feb115f1d32f12f
parsl.dataflow.memoization INFO: Task 310 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 489
parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/19e593de-557a-4ee4-b389-d28b2d53cc5e_characterizeImage_855911_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/19e593de-557a-4ee4-b389-d28b2d53cc5e_characterizeImage_855911_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/1e9c58f7-eb17-4d1e-9098-e2e8a3a5015d_isr_855911_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/1e9c58f7-eb17-4d1e-9098-e2e8a3a5015d_isr_855911_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:43 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 22f682f6ab6ce5b820b46effcd8e74db with result from task 351
parsl.dataflow.dflow DEBUG: Task 351 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 352 has memoization hash e65a926e2546ddee1fcfbd7a42239eaf
parsl.dataflow.memoization INFO: Task 352 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16abb9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 490
parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/f86520ac-ffae-4999-873d-215f504c9fa4_characterizeImage_855821_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/f86520ac-ffae-4999-873d-215f504c9fa4_characterizeImage_855821_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/a0fad8ea-dd3d-4057-a423-fb2f072527d3_isr_855821_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/a0fad8ea-dd3d-4057-a423-fb2f072527d3_isr_855821_21.stderr
parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d1318cf2f6aa6f6e5e9c72573b1a648f with result from task 339
parsl.dataflow.dflow DEBUG: Task 339 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 340 has memoization hash 65202395c5ad350c78c3dab603060924
parsl.dataflow.memoization INFO: Task 340 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16abad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 491
parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/a3bc10e0-15bf-47b1-945d-2986331c7274_characterizeImage_855821_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/a3bc10e0-15bf-47b1-945d-2986331c7274_characterizeImage_855821_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/ec877263-403d-4fd7-bfad-453559fc1baf_isr_855821_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/ec877263-403d-4fd7-bfad-453559fc1baf_isr_855821_13.stderr
parsl.dataflow.dflow INFO: Task 642 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2d7000ff65c56ae5e46bd130b33a9462 with result from task 642
parsl.dataflow.dflow DEBUG: Task 642 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 643 has memoization hash d8f5a9e56d413e290af63d6228ed3e6e
parsl.dataflow.memoization INFO: Task 643 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 643 try 0 launched on executor multi with executor id 492
parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/62f0adef-d30a-4385-a601-fac321ed6719_characterizeImage_855822_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/62f0adef-d30a-4385-a601-fac321ed6719_characterizeImage_855822_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/07bd7b60-05ee-4d79-8346-43e0b2a830b0_isr_855822_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/07bd7b60-05ee-4d79-8346-43e0b2a830b0_isr_855822_45.stderr
parsl.dataflow.dflow INFO: Task 498 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d249aa45ed05225f299821a449fa1db6 with result from task 498
parsl.dataflow.dflow DEBUG: Task 498 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 499 has memoization hash de6438478a8e43846cce1fba58d32119
parsl.dataflow.memoization INFO: Task 499 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159453a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 499 try 0 launched on executor multi with executor id 493
parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/a0381633-859b-4399-b7aa-962f9fe790f8_characterizeImage_855909_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/a0381633-859b-4399-b7aa-962f9fe790f8_characterizeImage_855909_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/9a2fc79b-09e7-4e4e-b4f1-bd6fc7199917_isr_855909_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/9a2fc79b-09e7-4e4e-b4f1-bd6fc7199917_isr_855909_20.stderr
parsl.dataflow.dflow INFO: Task 639 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bfd7cb72c92feddf9d64ca642ae6cd9f with result from task 639
parsl.dataflow.dflow DEBUG: Task 639 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 640 has memoization hash b82150ed2b6c8575218c021fafb8c554
parsl.dataflow.memoization INFO: Task 640 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 640 try 0 launched on executor multi with executor id 494
parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/87fffcf9-43c1-4b40-8db9-3ecba65a35b5_characterizeImage_855910_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/87fffcf9-43c1-4b40-8db9-3ecba65a35b5_characterizeImage_855910_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/af248e94-7a3c-45d1-9f37-8e9bf35355c5_isr_855910_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/af248e94-7a3c-45d1-9f37-8e9bf35355c5_isr_855910_25.stderr
parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2c3ee8ace56d6227c11044322f428631 with result from task 291
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:44 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow DEBUG: Task 291 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 292 has memoization hash 3c0b44ad919eed72c767095649f49e1a
parsl.dataflow.memoization INFO: Task 292 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 495
parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/ef9a9b17-4054-4ff4-9f53-6d3528930de5_characterizeImage_855911_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/ef9a9b17-4054-4ff4-9f53-6d3528930de5_characterizeImage_855911_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/31f20752-eeeb-4be3-8ee0-b2ccd9911b1d_isr_855911_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/31f20752-eeeb-4be3-8ee0-b2ccd9911b1d_isr_855911_14.stderr
parsl.dataflow.dflow INFO: Task 969 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 18d45166b33588a54a3a26eea2d282a5 with result from task 969
parsl.dataflow.dflow DEBUG: Task 969 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 970 has memoization hash e49f48cdbbe775f891a411364cffb819
parsl.dataflow.memoization INFO: Task 970 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 970 try 0 launched on executor multi with executor id 496
parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8ed5ca7a-d316-45a9-9282-658561e3897d_characterizeImage_855911_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8ed5ca7a-d316-45a9-9282-658561e3897d_characterizeImage_855911_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/5f3e9e52-4c84-40ce-8b34-64154b79787e_isr_855911_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/5f3e9e52-4c84-40ce-8b34-64154b79787e_isr_855911_48.stderr
parsl.dataflow.dflow INFO: Task 381 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2fe727447528e91b071031de3ec7543d with result from task 381
parsl.dataflow.dflow DEBUG: Task 381 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 382 has memoization hash cf3065ae5f0e177351fb8bb10e157752
parsl.dataflow.memoization INFO: Task 382 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 497
parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9385d400-89ab-4c09-9287-eb72b6bb36a3_characterizeImage_855910_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9385d400-89ab-4c09-9287-eb72b6bb36a3_characterizeImage_855910_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/a9432ef0-0898-4bb6-822a-1d7b0d9a1d1f_isr_855910_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/a9432ef0-0898-4bb6-822a-1d7b0d9a1d1f_isr_855910_33.stderr
parsl.dataflow.dflow INFO: Task 645 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e1e201db567655dfcdfad39fb3c17118 with result from task 645
parsl.dataflow.dflow DEBUG: Task 645 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 646 has memoization hash 90879420c427ddffa54bc42d8b0c7e34
parsl.dataflow.memoization INFO: Task 646 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 646 try 0 launched on executor multi with executor id 498
parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/7e2824a9-eef9-4c96-8626-11c7622aedd3_characterizeImage_855909_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/7e2824a9-eef9-4c96-8626-11c7622aedd3_characterizeImage_855909_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/91cbe4b7-1b4d-4e21-a9d3-6a70c3ebe35e_isr_855909_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/91cbe4b7-1b4d-4e21-a9d3-6a70c3ebe35e_isr_855909_56.stderr
parsl.dataflow.dflow INFO: Task 507 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0240ae59adae360d9068445325303d30 with result from task 507
parsl.dataflow.dflow DEBUG: Task 507 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 508 has memoization hash e471830c5441b3764b66123861fa01c4
parsl.dataflow.memoization INFO: Task 508 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15945d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 508 try 0 launched on executor multi with executor id 499
parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/c8fed829-a540-458f-9c63-db2f23d6a6fa_characterizeImage_855822_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/c8fed829-a540-458f-9c63-db2f23d6a6fa_characterizeImage_855822_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/f3e8d340-cac8-4010-84ce-817b2694a2f4_isr_855822_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/f3e8d340-cac8-4010-84ce-817b2694a2f4_isr_855822_17.stderr
parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 400fee910008cde85be72ef6c2e0189b with result from task 363
parsl.dataflow.dflow DEBUG: Task 363 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 364 has memoization hash f4c0a937b2c8a4a5092a6b0aebf15ce2
parsl.dataflow.memoization INFO: Task 364 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 500
parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/0488fa05-2443-4276-8395-c08857e44fb6_characterizeImage_855911_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/0488fa05-2443-4276-8395-c08857e44fb6_characterizeImage_855911_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/c6a84b29-e9ae-4f11-beab-1171a7b87217_isr_855911_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/c6a84b29-e9ae-4f11-beab-1171a7b87217_isr_855911_16.stderr
parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1f6b448f174fe448f900696e2678ad19 with result from task 327
parsl.dataflow.dflow DEBUG: Task 327 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 328 has memoization hash 32413be48fa0fc7becb3f0064b213c09
parsl.dataflow.memoization INFO: Task 328 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16aba0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 501
parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/9b622d83-8f4e-40ef-a3db-f40da51be251_characterizeImage_855911_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/9b622d83-8f4e-40ef-a3db-f40da51be251_characterizeImage_855911_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/de7d2dba-dced-4027-81bf-f24e47f5c104_isr_855911_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/de7d2dba-dced-4027-81bf-f24e47f5c104_isr_855911_31.stderr
parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry de7c4bc6cf97ad80281dd0a36f9a3420 with result from task 360
parsl.dataflow.dflow DEBUG: Task 360 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 361 has memoization hash 2eed5588475ea93ca17ba4e3a579a5a2
parsl.dataflow.memoization INFO: Task 361 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 502
parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/5113f289-949d-4d13-9b8f-7ec2c36a885f_characterizeImage_855821_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/5113f289-949d-4d13-9b8f-7ec2c36a885f_characterizeImage_855821_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/e859f7bc-52d0-47a5-82f8-3214362b2563_isr_855821_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/e859f7bc-52d0-47a5-82f8-3214362b2563_isr_855821_7.stderr
parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cbf94cd25cdacd1d7b68f370a2f1119c with result from task 318
parsl.dataflow.dflow DEBUG: Task 318 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 54924bd8e85d772764e81ae558f96a49
parsl.dataflow.memoization INFO: Task 319 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 503
parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/5b315c1d-f32c-4cde-830c-d6a7cb816ed6_characterizeImage_855909_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/5b315c1d-f32c-4cde-830c-d6a7cb816ed6_characterizeImage_855909_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/c4ff138e-90b3-4230-b9a0-7ce78fcc90b2_isr_855909_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/c4ff138e-90b3-4230-b9a0-7ce78fcc90b2_isr_855909_36.stderr
parsl.dataflow.dflow INFO: Task 387 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ad40134f38727f8bb8fd8e3d6a5f5ed7 with result from task 387
parsl.dataflow.dflow DEBUG: Task 387 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 388 has memoization hash 3780e9a62780018fc84e2f249e0c7966
parsl.dataflow.memoization INFO: Task 388 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 504
parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/6b2f241f-1025-4212-9ddb-6e251f4d1991_characterizeImage_855909_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/6b2f241f-1025-4212-9ddb-6e251f4d1991_characterizeImage_855909_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/b6586e48-4bfd-4638-b7e7-f6519a90d910_isr_855909_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/b6586e48-4bfd-4638-b7e7-f6519a90d910_isr_855909_46.stderr
parsl.dataflow.dflow INFO: Task 705 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ee92226c1f216372a44a5eefde20580b with result from task 705
parsl.dataflow.dflow DEBUG: Task 705 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 706 has memoization hash 5be39bcd95a42545a73cfd57ae88775a
parsl.dataflow.memoization INFO: Task 706 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 706 try 0 launched on executor multi with executor id 505
parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c828e551-6af1-4f5a-bef9-12d0f23ab841_characterizeImage_855820_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c828e551-6af1-4f5a-bef9-12d0f23ab841_characterizeImage_855820_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/39beb8d6-9d7c-4a69-b9ea-1b9189f489e3_isr_855820_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/39beb8d6-9d7c-4a69-b9ea-1b9189f489e3_isr_855820_23.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:45 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 471 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5d5104c7ea077fd7ef3b799f0886d538 with result from task 471
parsl.dataflow.dflow DEBUG: Task 471 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 472 has memoization hash 8fddaa93043b689d22af16fb82a8deab
parsl.dataflow.memoization INFO: Task 472 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 472 try 0 launched on executor multi with executor id 506
parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ebe408bc-8c11-4fc1-b73a-b70a37cd4cb3_characterizeImage_855820_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ebe408bc-8c11-4fc1-b73a-b70a37cd4cb3_characterizeImage_855820_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/fd8bf559-d120-4d3f-99bf-46d5e269c954_isr_855820_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/fd8bf559-d120-4d3f-99bf-46d5e269c954_isr_855820_5.stderr
parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5f0a512231637ba25b0be0b385ba80cc with result from task 288
parsl.dataflow.dflow DEBUG: Task 288 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 289 has memoization hash eb0b0832480e71dccc8f3c5c2caa1d82
parsl.dataflow.memoization INFO: Task 289 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 507
parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/2c6a2059-6b07-40fc-af84-7055d7481011_characterizeImage_855909_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/2c6a2059-6b07-40fc-af84-7055d7481011_characterizeImage_855909_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/245ac0e3-5bbb-465c-ac89-8875f42abf4c_isr_855909_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/245ac0e3-5bbb-465c-ac89-8875f42abf4c_isr_855909_11.stderr
parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 400244d7102faf5c01bda8895c361a18 with result from task 345
parsl.dataflow.dflow DEBUG: Task 345 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 346 has memoization hash 0c1a81737dba7926fed2c9acca6004e9
parsl.dataflow.memoization INFO: Task 346 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16abb380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 508
parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7e0bf724-a995-4352-b9f6-334d12aed276_characterizeImage_855910_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7e0bf724-a995-4352-b9f6-334d12aed276_characterizeImage_855910_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/7e9df86d-965c-4472-87bb-5b8650f28f02_isr_855910_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/7e9df86d-965c-4472-87bb-5b8650f28f02_isr_855910_27.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:45 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 648 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 91494feb39ecd9b5ea8144b906acda99 with result from task 648
parsl.dataflow.dflow DEBUG: Task 648 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 649 has memoization hash 6c16083a6bc22a33f459496b5e9d982d
parsl.dataflow.memoization INFO: Task 649 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 649 try 0 launched on executor multi with executor id 509
parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/b86f8dcf-7ba5-4429-b66e-65fba80f6930_characterizeImage_855911_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/b86f8dcf-7ba5-4429-b66e-65fba80f6930_characterizeImage_855911_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/99ed01cf-9567-4a33-b13e-83ffccfd02dc_isr_855911_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/99ed01cf-9567-4a33-b13e-83ffccfd02dc_isr_855911_12.stderr
parsl.dataflow.dflow INFO: Task 678 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 810b004fa849cacf2639f85473528862 with result from task 678
parsl.dataflow.dflow DEBUG: Task 678 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 679 has memoization hash d15e453695bd70ab742b3a8d374355ed
parsl.dataflow.memoization INFO: Task 679 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15808fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 679 try 0 launched on executor multi with executor id 510
parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/366329cc-e8f6-4267-8900-06f4967cc695_characterizeImage_855911_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/366329cc-e8f6-4267-8900-06f4967cc695_characterizeImage_855911_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/3ed9417e-f2eb-4b0f-aae6-3a810f150c84_isr_855911_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/3ed9417e-f2eb-4b0f-aae6-3a810f150c84_isr_855911_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:46 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cce3fda8964747abc04d0f32ebcca3e5 with result from task 315
parsl.dataflow.dflow DEBUG: Task 315 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 9ae71d5c76775b5a9b30f2124ef68db4
parsl.dataflow.memoization INFO: Task 316 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 511
parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/d12e0494-90db-4918-b785-528f128fdaa3_characterizeImage_855822_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/d12e0494-90db-4918-b785-528f128fdaa3_characterizeImage_855822_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/138afd5f-3c6e-482e-955b-30a436ede0e9_isr_855822_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/138afd5f-3c6e-482e-955b-30a436ede0e9_isr_855822_1.stderr
parsl.dataflow.dflow INFO: Task 1002 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9e888f37e5a91fcb9e4e289688d83145 with result from task 1002
parsl.dataflow.dflow DEBUG: Task 1002 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1003 has memoization hash 01652f8125662f1ed1978148f5f34d7b
parsl.dataflow.memoization INFO: Task 1003 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1003 try 0 launched on executor multi with executor id 512
parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/5e3181f6-b96b-42fe-bdd9-398afb2e2852_characterizeImage_855911_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/5e3181f6-b96b-42fe-bdd9-398afb2e2852_characterizeImage_855911_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/8443acdf-d0ab-4a16-957c-db4efb471e67_isr_855911_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/8443acdf-d0ab-4a16-957c-db4efb471e67_isr_855911_32.stderr
parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2bc071bf1a7237b43b84d13c93b5e0a1 with result from task 324
parsl.dataflow.dflow DEBUG: Task 324 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 07f04ed70709f8032682cf9daa92e3db
parsl.dataflow.memoization INFO: Task 325 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 513
parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c665b36c-f2f1-4331-99fc-e27f91d5f12e_characterizeImage_855911_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c665b36c-f2f1-4331-99fc-e27f91d5f12e_characterizeImage_855911_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a1f6eb89-6b84-4d84-b187-895c699c561f_isr_855911_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a1f6eb89-6b84-4d84-b187-895c699c561f_isr_855911_61.stderr
parsl.dataflow.dflow INFO: Task 819 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f657b853ffcd0730dbdbb501d803983f with result from task 819
parsl.dataflow.dflow DEBUG: Task 819 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 820 has memoization hash 8a07c89658bd0f5a7684eb22c162053c
parsl.dataflow.memoization INFO: Task 820 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15896340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 820 try 0 launched on executor multi with executor id 514
parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/b0348936-70a0-4820-b045-35897808d247_characterizeImage_855911_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/b0348936-70a0-4820-b045-35897808d247_characterizeImage_855911_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/5863efc6-5e06-4718-b16b-48617da16d2d_isr_855911_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/5863efc6-5e06-4718-b16b-48617da16d2d_isr_855911_50.stderr
parsl.dataflow.dflow INFO: Task 762 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3dd28856e9aa44e8a23b1e5ef70cca19 with result from task 762
parsl.dataflow.dflow DEBUG: Task 762 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 763 has memoization hash 0d6526c83493d3169a5daf6d48f0a3a9
parsl.dataflow.memoization INFO: Task 763 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 763 try 0 launched on executor multi with executor id 515
parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/6fc25a6f-6e01-4365-b3d2-a1a5e48c5ed9_characterizeImage_855821_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/6fc25a6f-6e01-4365-b3d2-a1a5e48c5ed9_characterizeImage_855821_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/3619b24e-5d96-4913-9877-cf3d4700d511_isr_855821_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/3619b24e-5d96-4913-9877-cf3d4700d511_isr_855821_2.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:46 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:46 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c360d30e247b60dbd82ab9baced6e43d with result from task 366
parsl.dataflow.dflow DEBUG: Task 366 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 367 has memoization hash 9c7f41f52ab45dd3052e1dba5f6a010b
parsl.dataflow.memoization INFO: Task 367 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 516
parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/703da37d-3fca-4755-b3e3-6aa0828a9ba7_characterizeImage_855909_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/703da37d-3fca-4755-b3e3-6aa0828a9ba7_characterizeImage_855909_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/89431413-bc51-4fcb-8471-a763da5c3c5d_isr_855909_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/89431413-bc51-4fcb-8471-a763da5c3c5d_isr_855909_38.stderr
parsl.dataflow.dflow INFO: Task 999 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9585ffb3f854db08a9344efe7c95ff67 with result from task 999
parsl.dataflow.dflow DEBUG: Task 999 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1000 has memoization hash 4a708be4d773e5e50e8c2eb2483ce45c
parsl.dataflow.memoization INFO: Task 1000 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1000 try 0 launched on executor multi with executor id 517
parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/3c4fcba1-3793-492e-8796-74c484951c00_characterizeImage_855911_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/3c4fcba1-3793-492e-8796-74c484951c00_characterizeImage_855911_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/8044c1ac-9ea3-40dc-9148-f3006ec9c833_isr_855911_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/8044c1ac-9ea3-40dc-9148-f3006ec9c833_isr_855911_35.stderr
parsl.dataflow.dflow INFO: Task 732 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d7019221d6788f0e057116089ab83088 with result from task 732
parsl.dataflow.dflow DEBUG: Task 732 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 733 has memoization hash 133780d1008f9ef703c15735db3a5fdc
parsl.dataflow.memoization INFO: Task 733 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15858860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 733 try 0 launched on executor multi with executor id 518
parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ab459aa0-b32a-44ff-8287-fc6c65ff6421_characterizeImage_855820_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ab459aa0-b32a-44ff-8287-fc6c65ff6421_characterizeImage_855820_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0bb11aa8-81cc-452c-b040-ef15ace97434_isr_855820_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0bb11aa8-81cc-452c-b040-ef15ace97434_isr_855820_30.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:47 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 792 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2e954ad5dcb2e382f78aa3b532423eba with result from task 792
parsl.dataflow.dflow DEBUG: Task 792 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 793 has memoization hash ab743d0a9411a9bc83138a49c9fe9623
parsl.dataflow.memoization INFO: Task 793 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15894720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 793 try 0 launched on executor multi with executor id 519
parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/3c16f8fe-60c8-44fc-979e-75cc5f8ebe10_characterizeImage_855911_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/3c16f8fe-60c8-44fc-979e-75cc5f8ebe10_characterizeImage_855911_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/7a549c9a-8980-4e2e-9b0b-afa1ab105449_isr_855911_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/7a549c9a-8980-4e2e-9b0b-afa1ab105449_isr_855911_40.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:47 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:47 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 636 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f82b7e423647e0e79b3c452353fcdb31 with result from task 636
parsl.dataflow.dflow DEBUG: Task 636 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 637 has memoization hash 4962ac62aa8a84962ab2f433f0ed4efd
parsl.dataflow.memoization INFO: Task 637 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 637 try 0 launched on executor multi with executor id 520
parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/fba0a16e-9a93-448b-a2f6-9940048ff6d5_characterizeImage_855911_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/fba0a16e-9a93-448b-a2f6-9940048ff6d5_characterizeImage_855911_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/1e0e4c2a-bf53-449f-ad6e-953b69e6b76d_isr_855911_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/1e0e4c2a-bf53-449f-ad6e-953b69e6b76d_isr_855911_49.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:48 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 972 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 077cacdf3fb1c0581b3e9e46a87aef50 with result from task 972
parsl.dataflow.dflow DEBUG: Task 972 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 973 has memoization hash 79effe03b3acbf8ababb7147bb9997aa
parsl.dataflow.memoization INFO: Task 973 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 973 try 0 launched on executor multi with executor id 521
parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/2952b10e-3da7-4654-b616-2753c65d6f46_characterizeImage_855821_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/2952b10e-3da7-4654-b616-2753c65d6f46_characterizeImage_855821_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/8e3addf9-7afa-4268-a9d2-a609ae9c4ba2_isr_855821_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/8e3addf9-7afa-4268-a9d2-a609ae9c4ba2_isr_855821_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:48 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 238 try 0 failed
parsl.dataflow.dflow INFO: Task 238 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/8b099f98-e72e-49c4-a069-956186a7ca4b_characterizeImage_855820_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/8b099f98-e72e-49c4-a069-956186a7ca4b_characterizeImage_855820_2.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 2929d5ef1b94dfc00498f4de6509140c
parsl.dataflow.memoization INFO: Task 238 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 238 try 1 launched on executor multi with executor id 522
parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/8b099f98-e72e-49c4-a069-956186a7ca4b_characterizeImage_855820_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/8b099f98-e72e-49c4-a069-956186a7ca4b_characterizeImage_855820_2.stderr
parsl.dataflow.dflow INFO: Task 1104 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e714b37d17002eebe1db94bd02d860df with result from task 1104
parsl.dataflow.dflow DEBUG: Task 1104 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1105 has memoization hash f4e736cebf1978a7b05b4b85446d4a4c
parsl.dataflow.memoization INFO: Task 1105 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1560cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1105 try 0 launched on executor multi with executor id 523
parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/34a4e514-e396-44a4-8579-280c35a0e37e_characterizeImage_855910_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/34a4e514-e396-44a4-8579-280c35a0e37e_characterizeImage_855910_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/b7beb70e-baa8-4ce1-a755-a20d4988b0ad_isr_855910_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/b7beb70e-baa8-4ce1-a755-a20d4988b0ad_isr_855910_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:49 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 396 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ff4776cbad219c6f8774c535739b6ed7 with result from task 396
parsl.dataflow.dflow DEBUG: Task 396 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 397 has memoization hash 4330d1e9bdd644c0cfecd4bfb85a7832
parsl.dataflow.memoization INFO: Task 397 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 524
parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/3cc8e705-4823-4be1-b464-b5a98497cfa8_characterizeImage_855820_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/3cc8e705-4823-4be1-b464-b5a98497cfa8_characterizeImage_855820_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/baee1d21-bcbf-463d-ba73-9038d1d0e0f9_isr_855820_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/baee1d21-bcbf-463d-ba73-9038d1d0e0f9_isr_855820_45.stderr
parsl.dataflow.dflow INFO: Task 390 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c5746101d528f11bf6023f25e4713ad5 with result from task 390
parsl.dataflow.dflow DEBUG: Task 390 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 391 has memoization hash 2d0bdfc1afe841916ee778fc7cd84eff
parsl.dataflow.memoization INFO: Task 391 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 525
parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/6bdfbd99-1c0b-4835-b6dd-ea609699059a_characterizeImage_855822_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/6bdfbd99-1c0b-4835-b6dd-ea609699059a_characterizeImage_855822_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/479c12f3-4a9b-4d31-b0c1-9d40260ce965_isr_855822_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/479c12f3-4a9b-4d31-b0c1-9d40260ce965_isr_855822_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:49 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:49 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:49 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 393 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 14bb492a538bbf169ef54bb7be75df38 with result from task 393
parsl.dataflow.dflow DEBUG: Task 393 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 394 has memoization hash 47f6bc9c712806bcec4947f6c22e9aac
parsl.dataflow.memoization INFO: Task 394 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 526
parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/ae40b21c-156a-4a29-a615-7f5c6496433e_characterizeImage_855822_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/ae40b21c-156a-4a29-a615-7f5c6496433e_characterizeImage_855822_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/61ab5a2d-d41b-4fc6-bae1-b04e9f849cc7_isr_855822_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/61ab5a2d-d41b-4fc6-bae1-b04e9f849cc7_isr_855822_49.stderr
parsl.dataflow.dflow INFO: Task 1101 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 69735f6ea01adcc496ecec1cfa6653ea with result from task 1101
parsl.dataflow.dflow DEBUG: Task 1101 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1102 has memoization hash 60efd1fff1d693dab1cabc7979677329
parsl.dataflow.memoization INFO: Task 1102 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1560ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1102 try 0 launched on executor multi with executor id 527
parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/5d35ab60-acd1-4811-8b9d-42f9098c6a10_characterizeImage_855820_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/5d35ab60-acd1-4811-8b9d-42f9098c6a10_characterizeImage_855820_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/2ab875e5-4a08-4db0-861e-4e39f1c39a58_isr_855820_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/2ab875e5-4a08-4db0-861e-4e39f1c39a58_isr_855820_51.stderr
parsl.dataflow.dflow INFO: Task 789 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f2ef1f2feb4d61ffbc590d35205b0ca9 with result from task 789
parsl.dataflow.dflow DEBUG: Task 789 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 790 has memoization hash 6174d350751e8537129f680a7b25bb79
parsl.dataflow.memoization INFO: Task 790 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15894400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 790 try 0 launched on executor multi with executor id 528
parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/cf3d7b4a-b9c9-4c50-b5fd-d3626fc8f9be_characterizeImage_855910_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/cf3d7b4a-b9c9-4c50-b5fd-d3626fc8f9be_characterizeImage_855910_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/aa55c955-2470-49d5-bc6e-1cfaab428703_isr_855910_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/aa55c955-2470-49d5-bc6e-1cfaab428703_isr_855910_53.stderr
parsl.dataflow.dflow INFO: Task 408 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 11760383242ae118047b07f7758e1e40 with result from task 408
parsl.dataflow.dflow DEBUG: Task 408 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 409 has memoization hash 3338ff462c27bf0bbc30271fafdfa6c9
parsl.dataflow.memoization INFO: Task 409 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 529
parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/24f7d608-6781-4af4-b04b-1ac9318bdaee_characterizeImage_855911_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/24f7d608-6781-4af4-b04b-1ac9318bdaee_characterizeImage_855911_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/9dffee86-183f-4d65-a8ac-547374119fd1_isr_855911_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/9dffee86-183f-4d65-a8ac-547374119fd1_isr_855911_42.stderr
parsl.dataflow.dflow INFO: Task 402 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d7aeac09f50c01205239888d6fb0e924 with result from task 402
parsl.dataflow.dflow DEBUG: Task 402 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 403 has memoization hash 4e11f6cc38ba088acb1f87ae4c64ce91
parsl.dataflow.memoization INFO: Task 403 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 530
parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/763a5823-d0dd-40d9-b6ea-3b68059c081b_characterizeImage_855821_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/763a5823-d0dd-40d9-b6ea-3b68059c081b_characterizeImage_855821_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/56bcc41a-fdb0-4c81-b667-28c0cd9b46fc_isr_855821_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/56bcc41a-fdb0-4c81-b667-28c0cd9b46fc_isr_855821_9.stderr
parsl.dataflow.dflow INFO: Task 420 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a2f2b992e21ce3065b93a810807aa89a with result from task 420
parsl.dataflow.dflow DEBUG: Task 420 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 421 has memoization hash 001043b7ded6bce21265d32f56a0c359
parsl.dataflow.memoization INFO: Task 421 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 531
parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ad1108bd-1043-4d78-93cf-c556eb997ffc_characterizeImage_855821_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ad1108bd-1043-4d78-93cf-c556eb997ffc_characterizeImage_855821_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/7e1838a3-76d4-4487-9c5b-2ead372bc6ec_isr_855821_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/7e1838a3-76d4-4487-9c5b-2ead372bc6ec_isr_855821_35.stderr
parsl.dataflow.dflow INFO: Task 399 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fdca56dfa13bbdd850f8f00d674688f2 with result from task 399
parsl.dataflow.dflow DEBUG: Task 399 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 400 has memoization hash d02a813f9a20fff9d7387d89346a53a4
parsl.dataflow.memoization INFO: Task 400 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 532
parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/0acbe00e-077f-40d6-b46a-c81f36470ac7_characterizeImage_855822_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/0acbe00e-077f-40d6-b46a-c81f36470ac7_characterizeImage_855822_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/7729a492-8cc1-4f7b-a724-102c7f79271e_isr_855822_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/7729a492-8cc1-4f7b-a724-102c7f79271e_isr_855822_40.stderr
parsl.dataflow.dflow INFO: Task 405 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e9960369b5859263af8b9cc7f186acf0 with result from task 405
parsl.dataflow.dflow DEBUG: Task 405 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 406 has memoization hash ce9204ec298c50eec38aefd5606a1ce7
parsl.dataflow.memoization INFO: Task 406 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 533
parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/eb64cad1-af37-4541-872f-304a0c3b7ef7_characterizeImage_855911_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/eb64cad1-af37-4541-872f-304a0c3b7ef7_characterizeImage_855911_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/b6c5e662-93a1-4ebe-8edc-eb1bd66704fe_isr_855911_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/b6c5e662-93a1-4ebe-8edc-eb1bd66704fe_isr_855911_10.stderr
parsl.dataflow.dflow INFO: Task 417 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2137c0dfa2d6ed2291c7c92df78fdf31 with result from task 417
parsl.dataflow.dflow DEBUG: Task 417 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 418 has memoization hash 1cc81a05f69f9c1952a4bc7fbc0a7bd8
parsl.dataflow.memoization INFO: Task 418 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 534
parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/3c3d465f-c012-4488-a809-61169c68a510_characterizeImage_855909_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/3c3d465f-c012-4488-a809-61169c68a510_characterizeImage_855909_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/ce3a3275-820e-44d0-af01-2f4adb8d848b_isr_855909_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/ce3a3275-820e-44d0-af01-2f4adb8d848b_isr_855909_41.stderr
parsl.dataflow.dflow INFO: Task 414 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 768708276cf0d74fa130d2a455c5d7d9 with result from task 414
parsl.dataflow.dflow DEBUG: Task 414 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 415 has memoization hash 59e089908d170d746641af59df492171
parsl.dataflow.memoization INFO: Task 415 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 535
parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/060e94fe-621c-44b6-8142-678759c1f991_characterizeImage_855911_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/060e94fe-621c-44b6-8142-678759c1f991_characterizeImage_855911_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/d664ee8f-c140-4718-91e7-c1df9f0aea39_isr_855911_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/d664ee8f-c140-4718-91e7-c1df9f0aea39_isr_855911_7.stderr
parsl.dataflow.dflow INFO: Task 411 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 09e0671e2ff9791f9bb962d770341e5f with result from task 411
parsl.dataflow.dflow DEBUG: Task 411 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 412 has memoization hash 6f0c07b36a62ab9476a579a819a49ccc
parsl.dataflow.memoization INFO: Task 412 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 536
parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/3871eafb-8810-4b9a-9f79-889af4c4e439_characterizeImage_855820_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/3871eafb-8810-4b9a-9f79-889af4c4e439_characterizeImage_855820_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/721aa592-09ba-43a9-9cd3-8dee11095dc3_isr_855820_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/721aa592-09ba-43a9-9cd3-8dee11095dc3_isr_855820_52.stderr
parsl.dataflow.dflow INFO: Task 585 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b5faf1a46519b6f34e545dd418a3b485 with result from task 585
parsl.dataflow.dflow DEBUG: Task 585 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 586 has memoization hash ff9d451e097094c06231d369b94b6ddd
parsl.dataflow.memoization INFO: Task 586 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 586 try 0 launched on executor multi with executor id 537
parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/77181560-7c30-4500-84ad-9db9b5388a6d_characterizeImage_855909_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/77181560-7c30-4500-84ad-9db9b5388a6d_characterizeImage_855909_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/d1a5c6f9-f2a9-49ce-bcff-6308cb46858a_isr_855909_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/d1a5c6f9-f2a9-49ce-bcff-6308cb46858a_isr_855909_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:51 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 675 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c24ccbeb3e15ca90db20615ab390a6c0 with result from task 675
parsl.dataflow.dflow DEBUG: Task 675 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 676 has memoization hash 451812b1dab00a469a518791c02c0260
parsl.dataflow.memoization INFO: Task 676 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15808cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 676 try 0 launched on executor multi with executor id 538
parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/5eecd24c-477b-4bef-b2db-2e7313305e4e_characterizeImage_855911_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/5eecd24c-477b-4bef-b2db-2e7313305e4e_characterizeImage_855911_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/935eb0ec-9d94-489a-aa44-b15e717c0003_isr_855911_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/935eb0ec-9d94-489a-aa44-b15e717c0003_isr_855911_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:52 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:53 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 501 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f0a745be60295eaac07c0b1a09e6776b with result from task 501
parsl.dataflow.dflow DEBUG: Task 501 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 502 has memoization hash c1e40febcda5f859ac9b8ff6978d7a24
parsl.dataflow.memoization INFO: Task 502 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159456c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 502 try 0 launched on executor multi with executor id 539
parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c22bf194-67bd-4de3-b364-7fce9115fda1_characterizeImage_855820_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c22bf194-67bd-4de3-b364-7fce9115fda1_characterizeImage_855820_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/58fcaa76-93b1-4654-ac2c-e8a2980d00fa_isr_855820_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/58fcaa76-93b1-4654-ac2c-e8a2980d00fa_isr_855820_62.stderr
parsl.dataflow.dflow INFO: Task 504 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1d94b247be3567ec1236e3c7130dc348 with result from task 504
parsl.dataflow.dflow DEBUG: Task 504 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 505 has memoization hash 2861f5d17b9c04c054eb9c03a8b9ad9b
parsl.dataflow.memoization INFO: Task 505 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159459e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 505 try 0 launched on executor multi with executor id 540
parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/1804422f-3cef-4925-bb30-b7ec8d29e1cd_characterizeImage_855821_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/1804422f-3cef-4925-bb30-b7ec8d29e1cd_characterizeImage_855821_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/85ab90c7-ab5c-4a96-a1fc-e8792d265edf_isr_855821_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/85ab90c7-ab5c-4a96-a1fc-e8792d265edf_isr_855821_61.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:53 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 918 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e48c9ee7b2f122972f87e5201f762ae5 with result from task 918
parsl.dataflow.dflow DEBUG: Task 918 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 919 has memoization hash 9a6a16168c3637605d65882f5bcbf9c8
parsl.dataflow.memoization INFO: Task 919 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15728ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 919 try 0 launched on executor multi with executor id 541
parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/e1237aea-99a6-479d-8eb6-9bde68f1a5b1_characterizeImage_855822_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/e1237aea-99a6-479d-8eb6-9bde68f1a5b1_characterizeImage_855822_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/b73491e0-ebc1-40ac-a84d-7aded2e6b91e_isr_855822_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/b73491e0-ebc1-40ac-a84d-7aded2e6b91e_isr_855822_52.stderr
parsl.dataflow.dflow INFO: Task 438 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f77b478bbeb3659b10e76807d732a021 with result from task 438
parsl.dataflow.dflow DEBUG: Task 438 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 439 has memoization hash 7c881bb88c4ac7559a8802d20c12978c
parsl.dataflow.memoization INFO: Task 439 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 439 try 0 launched on executor multi with executor id 542
parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/b2845ea8-710e-49b7-88fb-9ff22b16b5ae_characterizeImage_855821_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/b2845ea8-710e-49b7-88fb-9ff22b16b5ae_characterizeImage_855821_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/e5fd3902-9ee5-4857-a222-0775a6053065_isr_855821_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/e5fd3902-9ee5-4857-a222-0775a6053065_isr_855821_24.stderr
parsl.dataflow.dflow INFO: Task 423 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 929893aee6c7a85a1546f0631ebb1fe2 with result from task 423
parsl.dataflow.dflow DEBUG: Task 423 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 424 has memoization hash c8b5f8c007053702a7087546998e82a3
parsl.dataflow.memoization INFO: Task 424 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 543
parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/772062d8-8c67-4a07-9acf-af46db68f112_characterizeImage_855821_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/772062d8-8c67-4a07-9acf-af46db68f112_characterizeImage_855821_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/6e9990bd-7623-4089-a7c6-0f0b446eaee3_isr_855821_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/6e9990bd-7623-4089-a7c6-0f0b446eaee3_isr_855821_6.stderr
parsl.dataflow.dflow INFO: Task 444 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a62fd2a0bc224227173fd3973d81f95b with result from task 444
parsl.dataflow.dflow DEBUG: Task 444 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 445 has memoization hash 28a01e50ac1a87c0285a631611ca328b
parsl.dataflow.memoization INFO: Task 445 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 445 try 0 launched on executor multi with executor id 544
parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ddf62b0d-6d11-4bd3-a5b9-475c51c7b346_characterizeImage_855821_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ddf62b0d-6d11-4bd3-a5b9-475c51c7b346_characterizeImage_855821_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/6f542525-2f85-48b4-905a-e327cc7555a5_isr_855821_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/6f542525-2f85-48b4-905a-e327cc7555a5_isr_855821_10.stderr
parsl.dataflow.dflow INFO: Task 432 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f95964de22098d4d523ada1a8db76d5a with result from task 432
parsl.dataflow.dflow DEBUG: Task 432 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 433 has memoization hash 2e7fffb5a14c6d781eac3540286967fc
parsl.dataflow.memoization INFO: Task 433 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 545
parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/8e1ac0e1-a94d-457a-91d1-a2be127f1a13_characterizeImage_855821_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/8e1ac0e1-a94d-457a-91d1-a2be127f1a13_characterizeImage_855821_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/6ddc11b8-6c3b-446b-b6b4-b193a67a4fd9_isr_855821_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/6ddc11b8-6c3b-446b-b6b4-b193a67a4fd9_isr_855821_20.stderr
parsl.dataflow.dflow INFO: Task 429 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6b310fd9f29c2eca3624b35a753ae66a with result from task 429
parsl.dataflow.dflow DEBUG: Task 429 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 430 has memoization hash 3b27aea4bd6c4c814410fb1834776b53
parsl.dataflow.memoization INFO: Task 430 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 546
parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7257da59-452a-496b-95ff-a7fe3d3cef53_characterizeImage_855910_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7257da59-452a-496b-95ff-a7fe3d3cef53_characterizeImage_855910_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/a7d6fb5c-17dd-4d51-b8db-70eef770d099_isr_855910_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/a7d6fb5c-17dd-4d51-b8db-70eef770d099_isr_855910_2.stderr
parsl.dataflow.dflow INFO: Task 459 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d3afe1404b7a10b6b2d2718fc18817cc with result from task 459
parsl.dataflow.dflow DEBUG: Task 459 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 460 has memoization hash 7b782907e42e21e0ead315ad87f57d71
parsl.dataflow.memoization INFO: Task 460 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 460 try 0 launched on executor multi with executor id 547
parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/4152262c-c5ad-42f2-a5c9-fa5bbc820ef0_characterizeImage_855820_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/4152262c-c5ad-42f2-a5c9-fa5bbc820ef0_characterizeImage_855820_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/77a69965-45ae-4860-81cb-a88277ccd85d_isr_855820_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/77a69965-45ae-4860-81cb-a88277ccd85d_isr_855820_15.stderr
parsl.dataflow.dflow INFO: Task 462 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 051fe7fae511c820b4ee6cc61700da81 with result from task 462
parsl.dataflow.dflow DEBUG: Task 462 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 463 has memoization hash 5ddfe85142383fc284fb4790b66df0d4
parsl.dataflow.memoization INFO: Task 463 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 463 try 0 launched on executor multi with executor id 548
parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/644a413f-2a07-4ae8-b7c0-8cc140b0e0d8_characterizeImage_855911_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/644a413f-2a07-4ae8-b7c0-8cc140b0e0d8_characterizeImage_855911_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/55ca4f2d-3d61-43ad-be1e-c2eeb94c6600_isr_855911_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/55ca4f2d-3d61-43ad-be1e-c2eeb94c6600_isr_855911_18.stderr
parsl.dataflow.dflow INFO: Task 468 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a4cbac9c39fa5d13129af4980dd0d0b6 with result from task 468
parsl.dataflow.dflow DEBUG: Task 468 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 469 has memoization hash 0e4b7b4aaaa9d6fbafdb9c3006f3a068
parsl.dataflow.memoization INFO: Task 469 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 469 try 0 launched on executor multi with executor id 549
parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/343947a3-4527-47ff-aa03-2352ef2e7258_characterizeImage_855822_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/343947a3-4527-47ff-aa03-2352ef2e7258_characterizeImage_855822_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/dda49111-6b68-4dd8-b454-33ef2a3f248e_isr_855822_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/dda49111-6b68-4dd8-b454-33ef2a3f248e_isr_855822_30.stderr
parsl.dataflow.dflow INFO: Task 465 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 83c13054b867ff1a540d4caa6e4aa2bd with result from task 465
parsl.dataflow.dflow DEBUG: Task 465 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 466 has memoization hash 2de54ea0cfc93a9e192d91b7c5163f14
parsl.dataflow.memoization INFO: Task 466 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 466 try 0 launched on executor multi with executor id 550
parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/032cca53-7839-4440-88cc-6efbdb62717c_characterizeImage_855820_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/032cca53-7839-4440-88cc-6efbdb62717c_characterizeImage_855820_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/590face4-215c-438c-9ee7-d19e59ebc708_isr_855820_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/590face4-215c-438c-9ee7-d19e59ebc708_isr_855820_16.stderr
parsl.dataflow.dflow INFO: Task 453 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e3b4b5994fc76700cb97d17c183cfc54 with result from task 453
parsl.dataflow.dflow DEBUG: Task 453 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 454 has memoization hash d14a158fbd35f7cc1689a53cbba7940c
parsl.dataflow.memoization INFO: Task 454 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 454 try 0 launched on executor multi with executor id 551
parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9aed73d0-a6ae-48dd-a921-fda9dfcd032f_characterizeImage_855910_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9aed73d0-a6ae-48dd-a921-fda9dfcd032f_characterizeImage_855910_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/0b5a5c15-d743-41e4-8c35-0c8ea255a445_isr_855910_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/0b5a5c15-d743-41e4-8c35-0c8ea255a445_isr_855910_23.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 208 try 0 failed
parsl.dataflow.dflow INFO: Task 208 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/1772a9f8-c709-426c-ab7c-2153ab24dcfb_characterizeImage_855911_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/1772a9f8-c709-426c-ab7c-2153ab24dcfb_characterizeImage_855911_4.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 77b95601efaab5aa94e501a23b281932
parsl.dataflow.memoization INFO: Task 208 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a66340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 208 try 1 launched on executor multi with executor id 552
parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/1772a9f8-c709-426c-ab7c-2153ab24dcfb_characterizeImage_855911_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/1772a9f8-c709-426c-ab7c-2153ab24dcfb_characterizeImage_855911_4.stderr
parsl.dataflow.dflow INFO: Task 450 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 928ca308b039504e531394453ac21932 with result from task 450
parsl.dataflow.dflow DEBUG: Task 450 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 451 has memoization hash 5b05181bf527c2b92093634902ebdb63
parsl.dataflow.memoization INFO: Task 451 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 451 try 0 launched on executor multi with executor id 553
parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/14ae06c2-d118-4820-9d9b-bde09099e8d9_characterizeImage_855820_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/14ae06c2-d118-4820-9d9b-bde09099e8d9_characterizeImage_855820_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/87fa736e-3a82-4669-ae46-dcb0bffc97fb_isr_855820_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/87fa736e-3a82-4669-ae46-dcb0bffc97fb_isr_855820_36.stderr
parsl.dataflow.dflow INFO: Task 456 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 616dbeefb5551946c298afbaa0ea422f with result from task 456
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:54 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow DEBUG: Task 456 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 457 has memoization hash 06544cafc52b6bc8433f4458aa751c7e
parsl.dataflow.memoization INFO: Task 457 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 457 try 0 launched on executor multi with executor id 554
parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/98ffe426-8edd-4eea-92b8-392dd8c1543d_characterizeImage_855822_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/98ffe426-8edd-4eea-92b8-392dd8c1543d_characterizeImage_855822_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/376b4276-92bc-402b-8936-5f4b08567a05_isr_855822_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/376b4276-92bc-402b-8936-5f4b08567a05_isr_855822_36.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 187 try 0 failed
parsl.dataflow.dflow INFO: Task 187 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/380f55c6-32ea-4c0c-96bb-f30cf90e608d_characterizeImage_855911_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/380f55c6-32ea-4c0c-96bb-f30cf90e608d_characterizeImage_855911_29.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 187 has memoization hash a7ee76acc3a5fd0fd879cd4fd7b422ab
parsl.dataflow.memoization INFO: Task 187 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a64d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 187 try 1 launched on executor multi with executor id 555
parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/380f55c6-32ea-4c0c-96bb-f30cf90e608d_characterizeImage_855911_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/380f55c6-32ea-4c0c-96bb-f30cf90e608d_characterizeImage_855911_29.stderr
parsl.dataflow.dflow INFO: Task 447 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9d6d7757848c1b4fc06fbe31dd1cbf53 with result from task 447
parsl.dataflow.dflow DEBUG: Task 447 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 448 has memoization hash 2831fd4190ee1972e6626dd7ef2b77a7
parsl.dataflow.memoization INFO: Task 448 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 448 try 0 launched on executor multi with executor id 556
parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/77e4b1b3-a421-4fdd-bdf7-ae6daab0006d_characterizeImage_855909_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/77e4b1b3-a421-4fdd-bdf7-ae6daab0006d_characterizeImage_855909_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/6c63572e-47c1-4049-a6ce-dd971070e8fe_isr_855909_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/6c63572e-47c1-4049-a6ce-dd971070e8fe_isr_855909_47.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:55 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 573 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 24afe80a1502f0af83c19bd2597d88c7 with result from task 573
parsl.dataflow.dflow DEBUG: Task 573 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 574 has memoization hash 004ab7dd17d5aab59251c0b13f3dbb86
parsl.dataflow.memoization INFO: Task 574 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 574 try 0 launched on executor multi with executor id 557
parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/0039363b-4812-4620-a5b0-0bc9d3b18d5d_characterizeImage_855822_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/0039363b-4812-4620-a5b0-0bc9d3b18d5d_characterizeImage_855822_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/ba8e7a7c-9ed0-4453-99b5-d5b28aa51155_isr_855822_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/ba8e7a7c-9ed0-4453-99b5-d5b28aa51155_isr_855822_16.stderr
parsl.dataflow.dflow INFO: Task 576 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cbf06a97908e6bda8beb3c53339e5f48 with result from task 576
parsl.dataflow.dflow DEBUG: Task 576 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 577 has memoization hash d72d77aeba683cb553260f47f0ca9d24
parsl.dataflow.memoization INFO: Task 577 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 577 try 0 launched on executor multi with executor id 558
parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b0af1003-e9b5-405e-9c1e-712df5a6a216_characterizeImage_855910_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b0af1003-e9b5-405e-9c1e-712df5a6a216_characterizeImage_855910_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/dbeb831e-9585-405f-8eab-4f2bb3bc60f2_isr_855910_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/dbeb831e-9585-405f-8eab-4f2bb3bc60f2_isr_855910_16.stderr
parsl.dataflow.dflow INFO: Task 579 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 09f289eb7001b49689d62b354e359dd2 with result from task 579
parsl.dataflow.dflow DEBUG: Task 579 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 580 has memoization hash 4a2f50a6ad030e9c91932dfe894d56bc
parsl.dataflow.memoization INFO: Task 580 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 580 try 0 launched on executor multi with executor id 559
parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/5ed7c968-5683-48f2-aa3a-07b8581d799f_characterizeImage_855822_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/5ed7c968-5683-48f2-aa3a-07b8581d799f_characterizeImage_855822_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/3689fdee-a1f6-44bc-af7f-b959edfa1e37_isr_855822_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/3689fdee-a1f6-44bc-af7f-b959edfa1e37_isr_855822_51.stderr
parsl.dataflow.dflow INFO: Task 570 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 23a95ebbe13d259c0bccea1e22ceafe6 with result from task 570
parsl.dataflow.dflow DEBUG: Task 570 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 571 has memoization hash cdac144bbb8353fc5cabbca7a2b0df76
parsl.dataflow.memoization INFO: Task 571 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 571 try 0 launched on executor multi with executor id 560
parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/d2050731-17bb-472a-a78e-a1f0abadb25f_characterizeImage_855911_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/d2050731-17bb-472a-a78e-a1f0abadb25f_characterizeImage_855911_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/cf0ef916-69b0-48b7-bbee-bc1666a1f629_isr_855911_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/cf0ef916-69b0-48b7-bbee-bc1666a1f629_isr_855911_39.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 199 try 0 failed
parsl.dataflow.dflow INFO: Task 199 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/ac9d00eb-08f8-4dba-acad-e1bb1800f245_characterizeImage_855910_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/ac9d00eb-08f8-4dba-acad-e1bb1800f245_characterizeImage_855910_15.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 199 has memoization hash a807ccd39a385f95f710aec7ff05a570
parsl.dataflow.memoization INFO: Task 199 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a659e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 199 try 1 launched on executor multi with executor id 561
parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/ac9d00eb-08f8-4dba-acad-e1bb1800f245_characterizeImage_855910_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/ac9d00eb-08f8-4dba-acad-e1bb1800f245_characterizeImage_855910_15.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 205 try 0 failed
parsl.dataflow.dflow INFO: Task 205 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a5fefbfe-5e7b-4e1a-976e-347e18a8306b_characterizeImage_855910_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a5fefbfe-5e7b-4e1a-976e-347e18a8306b_characterizeImage_855910_49.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 205 has memoization hash c4da951aec1cde9c135aa677eacde2e7
parsl.dataflow.memoization INFO: Task 205 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a66020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 205 try 1 launched on executor multi with executor id 562
parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a5fefbfe-5e7b-4e1a-976e-347e18a8306b_characterizeImage_855910_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a5fefbfe-5e7b-4e1a-976e-347e18a8306b_characterizeImage_855910_49.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:55 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 759 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0893d23a4d693cc559b7b989623dd33d with result from task 759
parsl.dataflow.dflow DEBUG: Task 759 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 760 has memoization hash 1a97784f91973dec931145df95f01d7f
parsl.dataflow.memoization INFO: Task 760 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 760 try 0 launched on executor multi with executor id 563
parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7194435b-70cf-4770-9caa-7e050318a1d4_characterizeImage_855910_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7194435b-70cf-4770-9caa-7e050318a1d4_characterizeImage_855910_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/2672b821-e17b-4ecb-b838-5e4577fdcf4b_isr_855910_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/2672b821-e17b-4ecb-b838-5e4577fdcf4b_isr_855910_30.stderr
parsl.dataflow.dflow INFO: Task 435 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f131ac97d2f442e4c8d79d56a63efbcd with result from task 435
parsl.dataflow.dflow DEBUG: Task 435 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 436 has memoization hash f6f93299db031d64f245bb505710a6ba
parsl.dataflow.memoization INFO: Task 436 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 436 try 0 launched on executor multi with executor id 564
parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/80762dfd-6863-4cbe-8166-6ec8cdc2870a_characterizeImage_855911_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/80762dfd-6863-4cbe-8166-6ec8cdc2870a_characterizeImage_855911_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/62f4a502-b4e1-4cab-a713-7912b372b5dd_isr_855911_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/62f4a502-b4e1-4cab-a713-7912b372b5dd_isr_855911_60.stderr
parsl.dataflow.dflow INFO: Task 582 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 082cc518801e3ed8e60522a01c48bc23 with result from task 582
parsl.dataflow.dflow DEBUG: Task 582 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 583 has memoization hash 7dce9f9958dbc16b0b7bd88dc7d7483b
parsl.dataflow.memoization INFO: Task 583 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 583 try 0 launched on executor multi with executor id 565
parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ffbbb12c-8d7c-48ea-a91f-14769608b6cc_characterizeImage_855820_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ffbbb12c-8d7c-48ea-a91f-14769608b6cc_characterizeImage_855820_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/be1f29ec-6848-43c1-82e8-7f3546e155f3_isr_855820_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/be1f29ec-6848-43c1-82e8-7f3546e155f3_isr_855820_48.stderr
parsl.dataflow.dflow INFO: Task 426 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 902324800f6f75a8dee6941a23dc3368 with result from task 426
parsl.dataflow.dflow DEBUG: Task 426 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 427 has memoization hash 6ea846b3acb4147e72347a118b3f47c6
parsl.dataflow.memoization INFO: Task 427 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 566
parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/7e91f3bc-72b0-4250-a184-f5c9225e1839_characterizeImage_855821_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/7e91f3bc-72b0-4250-a184-f5c9225e1839_characterizeImage_855821_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/1a9b071a-dbbb-46a2-81b2-0a264ad41635_isr_855821_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/1a9b071a-dbbb-46a2-81b2-0a264ad41635_isr_855821_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:56 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 567 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3fb230a6fe0e3c58b4c1cfcdac379ad3 with result from task 567
parsl.dataflow.dflow DEBUG: Task 567 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 568 has memoization hash 55b894b39f80d7c716557491f30be694
parsl.dataflow.memoization INFO: Task 568 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 568 try 0 launched on executor multi with executor id 567
parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/7886e349-ffb6-4da2-83fe-b94e2d0dd6c7_characterizeImage_855820_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/7886e349-ffb6-4da2-83fe-b94e2d0dd6c7_characterizeImage_855820_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/91c4d1d2-13ac-446e-822a-f87334ee487e_isr_855820_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/91c4d1d2-13ac-446e-822a-f87334ee487e_isr_855820_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:56 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 564 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4bfa315d06e36431b1c1ae6c547f8b70 with result from task 564
parsl.dataflow.dflow DEBUG: Task 564 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 565 has memoization hash d076223a307a029bebf539c147350d77
parsl.dataflow.memoization INFO: Task 565 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:57 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Parsl task 565 try 0 launched on executor multi with executor id 568
parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/9f7493a5-1ca2-4541-b84d-b795883c4ed8_characterizeImage_855822_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/9f7493a5-1ca2-4541-b84d-b795883c4ed8_characterizeImage_855822_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/d8083ccd-bcc1-482c-9eca-fc5f2a94aa78_isr_855822_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/d8083ccd-bcc1-482c-9eca-fc5f2a94aa78_isr_855822_53.stderr
parsl.dataflow.dflow INFO: Task 489 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6e80598e94f615aa6685c4e0677eafee with result from task 489
parsl.dataflow.dflow DEBUG: Task 489 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 490 has memoization hash 602f531ac7efb4981ef8c7009926e569
parsl.dataflow.memoization INFO: Task 490 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15944a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 490 try 0 launched on executor multi with executor id 569
parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/c980a735-aac0-4722-a98e-f4943b00848c_characterizeImage_855821_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/c980a735-aac0-4722-a98e-f4943b00848c_characterizeImage_855821_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/4c8a1b52-ccb1-4c0e-927a-8c1028160320_isr_855821_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/4c8a1b52-ccb1-4c0e-927a-8c1028160320_isr_855821_3.stderr
parsl.dataflow.dflow INFO: Task 486 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 970a58ad43b8cde17de85b6d1f947018 with result from task 486
parsl.dataflow.dflow DEBUG: Task 486 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 487 has memoization hash f0b387c827d3396f2ea83ddce491dd4e
parsl.dataflow.memoization INFO: Task 487 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15944720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 487 try 0 launched on executor multi with executor id 570
parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/fef6d108-d03d-454e-835e-e83cd212e5ed_characterizeImage_855909_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/fef6d108-d03d-454e-835e-e83cd212e5ed_characterizeImage_855909_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/af8f8a6d-df49-41ec-ac68-181705ad5401_isr_855909_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/af8f8a6d-df49-41ec-ac68-181705ad5401_isr_855909_16.stderr
parsl.dataflow.dflow INFO: Task 474 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d41380e772795a90bb355b9e3f340d94 with result from task 474
parsl.dataflow.dflow DEBUG: Task 474 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 475 has memoization hash ab15d2919918ea2d48c044d48c4d736c
parsl.dataflow.memoization INFO: Task 475 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 475 try 0 launched on executor multi with executor id 571
parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/8d33728a-05d1-4f47-b28c-12afb1775c2e_characterizeImage_855821_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/8d33728a-05d1-4f47-b28c-12afb1775c2e_characterizeImage_855821_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/b140b99f-6507-4d93-bba2-4a363339b6c8_isr_855821_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/b140b99f-6507-4d93-bba2-4a363339b6c8_isr_855821_59.stderr
parsl.dataflow.dflow INFO: Task 477 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9d4f8d0114f835545017e0284a39722a with result from task 477
parsl.dataflow.dflow DEBUG: Task 477 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 478 has memoization hash a23686468a4c2dcf39dfa301f924b081
parsl.dataflow.memoization INFO: Task 478 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 478 try 0 launched on executor multi with executor id 572
parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/787a3392-1133-43a9-aea9-8dc23a153ff9_characterizeImage_855820_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/787a3392-1133-43a9-aea9-8dc23a153ff9_characterizeImage_855820_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/a25ffa53-f7b0-460a-85c3-9451793e0317_isr_855820_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/a25ffa53-f7b0-460a-85c3-9451793e0317_isr_855820_42.stderr
parsl.dataflow.dflow INFO: Task 495 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cc583912eebf1c11135d30af840a671e with result from task 495
parsl.dataflow.dflow DEBUG: Task 495 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 496 has memoization hash db321908b613bd9b2ba56cf29fbc7d17
parsl.dataflow.memoization INFO: Task 496 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15945080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 496 try 0 launched on executor multi with executor id 573
parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/42ad4c79-eca6-4c87-a81b-9077416fbe9d_characterizeImage_855909_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/42ad4c79-eca6-4c87-a81b-9077416fbe9d_characterizeImage_855909_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/e7d9774b-1689-4e20-8ca0-4c5715afaa73_isr_855909_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/e7d9774b-1689-4e20-8ca0-4c5715afaa73_isr_855909_18.stderr
parsl.dataflow.dflow INFO: Task 492 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4261fa4f12168c01072720ac2fb87914 with result from task 492
parsl.dataflow.dflow DEBUG: Task 492 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 493 has memoization hash 5e75331aa359a1f8ab04cc0a973468c1
parsl.dataflow.memoization INFO: Task 493 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15944d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 493 try 0 launched on executor multi with executor id 574
parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/b09b9d40-9292-466c-9491-6721b6df807d_characterizeImage_855911_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/b09b9d40-9292-466c-9491-6721b6df807d_characterizeImage_855911_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/3d87f476-30a2-44df-a22d-bbf76d3c3c9e_isr_855911_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/3d87f476-30a2-44df-a22d-bbf76d3c3c9e_isr_855911_43.stderr
parsl.dataflow.dflow INFO: Task 561 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 95f8e9bffdeb25c1f063ab1d7f79fadd with result from task 561
parsl.dataflow.dflow DEBUG: Task 561 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 562 has memoization hash ad86e64ad4fed28b9d9c383cbd8b9d4b
parsl.dataflow.memoization INFO: Task 562 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 562 try 0 launched on executor multi with executor id 575
parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/87a40593-9647-4c1e-88d8-f549b330a5b5_characterizeImage_855821_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/87a40593-9647-4c1e-88d8-f549b330a5b5_characterizeImage_855821_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/87caa8a9-bf00-4bb2-8ed6-c603e21ed65a_isr_855821_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/87caa8a9-bf00-4bb2-8ed6-c603e21ed65a_isr_855821_55.stderr
parsl.dataflow.dflow INFO: Task 483 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3c2a78bfc09531b4eb4d2a244ca7469e with result from task 483
parsl.dataflow.dflow DEBUG: Task 483 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 484 has memoization hash f7db7810fdee511978569b5c27c71b33
parsl.dataflow.memoization INFO: Task 484 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15944400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 484 try 0 launched on executor multi with executor id 576
parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/895d27e2-90dc-4a78-80cc-6f08841898e2_characterizeImage_855820_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/895d27e2-90dc-4a78-80cc-6f08841898e2_characterizeImage_855820_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/6b4b6193-8c64-4a36-a782-94d5386be9c2_isr_855820_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/6b4b6193-8c64-4a36-a782-94d5386be9c2_isr_855820_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:57 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 441 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 81c254272fd08a7e4c883ade2351ae79 with result from task 441
parsl.dataflow.dflow DEBUG: Task 441 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 442 has memoization hash 44cd07ae95eee81d77182c83d3f9251c
parsl.dataflow.memoization INFO: Task 442 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 442 try 0 launched on executor multi with executor id 577
parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/bb1c1232-a3c9-4514-8fc9-3cb40eb30104_characterizeImage_855910_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/bb1c1232-a3c9-4514-8fc9-3cb40eb30104_characterizeImage_855910_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/d0804ff0-cb56-44d2-afdc-98bf98c36c99_isr_855910_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/d0804ff0-cb56-44d2-afdc-98bf98c36c99_isr_855910_60.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:58 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:58 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 328 try 0 failed
parsl.dataflow.dflow INFO: Task 328 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/9b622d83-8f4e-40ef-a3db-f40da51be251_characterizeImage_855911_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/9b622d83-8f4e-40ef-a3db-f40da51be251_characterizeImage_855911_31.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 328 has memoization hash 32413be48fa0fc7becb3f0064b213c09
parsl.dataflow.memoization INFO: Task 328 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16aba0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 328 try 1 launched on executor multi with executor id 578
parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/9b622d83-8f4e-40ef-a3db-f40da51be251_characterizeImage_855911_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/9b622d83-8f4e-40ef-a3db-f40da51be251_characterizeImage_855911_31.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:59 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 480 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b99aa0d4513c9c4bd467dc2ea54df649 with result from task 480
parsl.dataflow.dflow DEBUG: Task 480 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 481 has memoization hash eec4f4f8ca0671b2c4d3b328af5b25ce
parsl.dataflow.memoization INFO: Task 481 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 481 try 0 launched on executor multi with executor id 579
parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/0e819d5c-7227-4014-918a-76fdab33aba8_characterizeImage_855909_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/0e819d5c-7227-4014-918a-76fdab33aba8_characterizeImage_855909_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/3f7ed719-c565-4f1e-9134-103224edc290_isr_855909_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/3f7ed719-c565-4f1e-9134-103224edc290_isr_855909_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:55:59 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 663 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b81be6681b4e7ba740e90241f1f7062d with result from task 663
parsl.dataflow.dflow DEBUG: Task 663 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 664 has memoization hash 983270ae2cf3bab3fabbfa493f7cfb76
parsl.dataflow.memoization INFO: Task 664 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 664 try 0 launched on executor multi with executor id 580
parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/630877c9-a022-4ed5-97ac-e679b28da27f_characterizeImage_855910_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/630877c9-a022-4ed5-97ac-e679b28da27f_characterizeImage_855910_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/b91004a8-4e03-49c9-bdc4-405cb3cdbe3b_isr_855910_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/b91004a8-4e03-49c9-bdc4-405cb3cdbe3b_isr_855910_46.stderr
parsl.dataflow.dflow INFO: Task 666 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7b997c6a08b622bf86b2e9432594ce71 with result from task 666
parsl.dataflow.dflow DEBUG: Task 666 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 667 has memoization hash 78cf8cd67a0167e41241910c3a8c987b
parsl.dataflow.memoization INFO: Task 667 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158080e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 667 try 0 launched on executor multi with executor id 581
parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/4756df3f-6472-46f7-a6a8-45b5f6e1f99d_characterizeImage_855909_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/4756df3f-6472-46f7-a6a8-45b5f6e1f99d_characterizeImage_855909_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/16898732-a5dc-41d0-bcf0-15a9f84a964f_isr_855909_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/16898732-a5dc-41d0-bcf0-15a9f84a964f_isr_855909_33.stderr
parsl.dataflow.dflow INFO: Task 669 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3b499dcc083f40615155d5b1d440a7f1 with result from task 669
parsl.dataflow.dflow DEBUG: Task 669 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 670 has memoization hash 35dde2a4d9ccd4f23e5e0dfc62840a90
parsl.dataflow.memoization INFO: Task 670 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15808680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 670 try 0 launched on executor multi with executor id 582
parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/e25fe73f-595a-4f0f-8204-3dc027466c20_characterizeImage_855910_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/e25fe73f-595a-4f0f-8204-3dc027466c20_characterizeImage_855910_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/ed99eb83-f804-4f46-af1e-5c83c40aba8c_isr_855910_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/ed99eb83-f804-4f46-af1e-5c83c40aba8c_isr_855910_19.stderr
parsl.dataflow.dflow INFO: Task 660 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f09671bf22f6ff046a6e6af48b3d6254 with result from task 660
parsl.dataflow.dflow DEBUG: Task 660 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 661 has memoization hash 383d3883492a463a23a066bcfaf3b99e
parsl.dataflow.memoization INFO: Task 661 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 661 try 0 launched on executor multi with executor id 583
parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/aa3cef68-3e27-434f-97e3-def31bd73121_characterizeImage_855820_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/aa3cef68-3e27-434f-97e3-def31bd73121_characterizeImage_855820_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/778d4852-0f3b-4d0e-b09e-a2ebdc546423_isr_855820_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/778d4852-0f3b-4d0e-b09e-a2ebdc546423_isr_855820_44.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 202 try 0 failed
parsl.dataflow.dflow INFO: Task 202 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c5a4fe4e-d413-4790-808a-ade5a22926eb_characterizeImage_855911_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c5a4fe4e-d413-4790-808a-ade5a22926eb_characterizeImage_855911_2.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 202 has memoization hash c0d66659766a9cef44d1239f370f6c1f
parsl.dataflow.memoization INFO: Task 202 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a65d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 202 try 1 launched on executor multi with executor id 584
parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c5a4fe4e-d413-4790-808a-ade5a22926eb_characterizeImage_855911_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c5a4fe4e-d413-4790-808a-ade5a22926eb_characterizeImage_855911_2.stderr
parsl.dataflow.dflow INFO: Task 651 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c640c2c85c43eb654db99b71e4f34cc8 with result from task 651
parsl.dataflow.dflow DEBUG: Task 651 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 652 has memoization hash 02f1ce6cca98a69d02af6a11c05ac0fd
parsl.dataflow.memoization INFO: Task 652 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 652 try 0 launched on executor multi with executor id 585
parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/bf49286d-cee7-44a0-8ec6-b91587dabdec_characterizeImage_855820_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/bf49286d-cee7-44a0-8ec6-b91587dabdec_characterizeImage_855820_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/d13986ba-7ea0-4590-b5e2-8ca97fc487f5_isr_855820_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/d13986ba-7ea0-4590-b5e2-8ca97fc487f5_isr_855820_61.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 970 try 0 failed
parsl.dataflow.dflow INFO: Task 970 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8ed5ca7a-d316-45a9-9282-658561e3897d_characterizeImage_855911_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8ed5ca7a-d316-45a9-9282-658561e3897d_characterizeImage_855911_48.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 970 has memoization hash e49f48cdbbe775f891a411364cffb819
parsl.dataflow.memoization INFO: Task 970 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 970 try 1 launched on executor multi with executor id 586
parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8ed5ca7a-d316-45a9-9282-658561e3897d_characterizeImage_855911_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8ed5ca7a-d316-45a9-9282-658561e3897d_characterizeImage_855911_48.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:00 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 355 try 0 failed
parsl.dataflow.dflow INFO: Task 355 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/ab0f61a5-3ad2-4dec-8705-232145f6b45e_characterizeImage_855911_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/ab0f61a5-3ad2-4dec-8705-232145f6b45e_characterizeImage_855911_1.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 355 has memoization hash 8c591d801a49ded05e4053bf75bb6bff
parsl.dataflow.memoization INFO: Task 355 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16abbce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 355 try 1 launched on executor multi with executor id 587
parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/ab0f61a5-3ad2-4dec-8705-232145f6b45e_characterizeImage_855911_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/ab0f61a5-3ad2-4dec-8705-232145f6b45e_characterizeImage_855911_1.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 763 try 0 failed
parsl.dataflow.dflow INFO: Task 763 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/6fc25a6f-6e01-4365-b3d2-a1a5e48c5ed9_characterizeImage_855821_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/6fc25a6f-6e01-4365-b3d2-a1a5e48c5ed9_characterizeImage_855821_2.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 763 has memoization hash 0d6526c83493d3169a5daf6d48f0a3a9
parsl.dataflow.memoization INFO: Task 763 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 763 try 1 launched on executor multi with executor id 588
parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/6fc25a6f-6e01-4365-b3d2-a1a5e48c5ed9_characterizeImage_855821_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/6fc25a6f-6e01-4365-b3d2-a1a5e48c5ed9_characterizeImage_855821_2.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 238 try 1 failed
parsl.dataflow.dflow ERROR: Task 238 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 2929d5ef1b94dfc00498f4de6509140c with result from task 238
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 239 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 239 try 0 failed
parsl.dataflow.dflow INFO: Task 239 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/e3fd10ff-ea24-41ae-ae54-388ff6937a77_calibrate_855820_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/e3fd10ff-ea24-41ae-ae54-388ff6937a77_calibrate_855820_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/8b099f98-e72e-49c4-a069-956186a7ca4b_characterizeImage_855820_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/8b099f98-e72e-49c4-a069-956186a7ca4b_characterizeImage_855820_2.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:01 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 307 try 0 failed
parsl.dataflow.dflow INFO: Task 307 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9ed1bf3e-ea29-498b-8e9b-247ecfad18e0_characterizeImage_855910_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9ed1bf3e-ea29-498b-8e9b-247ecfad18e0_characterizeImage_855910_47.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 307 has memoization hash b638ea1d813b2bd661d044d5f67a3d62
parsl.dataflow.memoization INFO: Task 307 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 307 try 1 launched on executor multi with executor id 589
parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9ed1bf3e-ea29-498b-8e9b-247ecfad18e0_characterizeImage_855910_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9ed1bf3e-ea29-498b-8e9b-247ecfad18e0_characterizeImage_855910_47.stderr
parsl.dataflow.dflow INFO: Task 744 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e9026b88a57bc925b25ea1ff78426e85 with result from task 744
parsl.dataflow.dflow DEBUG: Task 744 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 745 has memoization hash 55666f5781a91b008ccc5454f21c0658
parsl.dataflow.memoization INFO: Task 745 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158594e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 745 try 0 launched on executor multi with executor id 590
parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/e7e5ef6d-6383-4cff-986a-abc4c3dc521b_characterizeImage_855909_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/e7e5ef6d-6383-4cff-986a-abc4c3dc521b_characterizeImage_855909_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/7536f64a-7e03-48a3-b1df-d3b43eba886a_isr_855909_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/7536f64a-7e03-48a3-b1df-d3b43eba886a_isr_855909_55.stderr
parsl.dataflow.dflow INFO: Task 702 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7e6b9442e0e8255d9e5ef5a3a8ed3328 with result from task 702
parsl.dataflow.dflow DEBUG: Task 702 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 703 has memoization hash 244c05ef383116b5b0934e69e10928ef
parsl.dataflow.memoization INFO: Task 703 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 703 try 0 launched on executor multi with executor id 591
parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/416f3cfe-711b-4d65-bc3c-7c415e030bfd_characterizeImage_855909_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/416f3cfe-711b-4d65-bc3c-7c415e030bfd_characterizeImage_855909_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/fc43ee7e-dd81-491c-8b4f-5456638e7c0f_isr_855909_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/fc43ee7e-dd81-491c-8b4f-5456638e7c0f_isr_855909_27.stderr
parsl.dataflow.dflow INFO: Task 681 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f9af664cfebd608ad192b5e6ec03748a with result from task 681
parsl.dataflow.dflow DEBUG: Task 681 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 682 has memoization hash 9f6b1600b6e9fa9a7b3a13478c5cffca
parsl.dataflow.memoization INFO: Task 682 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15809300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 682 try 0 launched on executor multi with executor id 592
parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/9f393b75-dbc6-46b2-b267-622feadcc00e_characterizeImage_855911_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/9f393b75-dbc6-46b2-b267-622feadcc00e_characterizeImage_855911_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/4f6ccac5-ea84-4018-b9b1-9821a726d447_isr_855911_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/4f6ccac5-ea84-4018-b9b1-9821a726d447_isr_855911_25.stderr
parsl.dataflow.dflow INFO: Task 696 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9065b6305d94d4a456ef2e0df3255936 with result from task 696
parsl.dataflow.dflow DEBUG: Task 696 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 697 has memoization hash bb0bc9d7594dc47381745ce0f6c10760
parsl.dataflow.memoization INFO: Task 697 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 697 try 0 launched on executor multi with executor id 593
parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/66ebc2c9-521d-4d35-a305-f74b9a19e580_characterizeImage_855910_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/66ebc2c9-521d-4d35-a305-f74b9a19e580_characterizeImage_855910_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/7f7f12fa-f45f-4ed6-942c-30a21b76465a_isr_855910_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/7f7f12fa-f45f-4ed6-942c-30a21b76465a_isr_855910_21.stderr
parsl.dataflow.dflow INFO: Task 699 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ca8395580f5d05902d6ca1b402b82dd7 with result from task 699
parsl.dataflow.dflow DEBUG: Task 699 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 700 has memoization hash 0f12d4053081539917dc37272cde2724
parsl.dataflow.memoization INFO: Task 700 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 700 try 0 launched on executor multi with executor id 594
parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/de38ec18-22eb-43af-ac56-74b362e327c4_characterizeImage_855821_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/de38ec18-22eb-43af-ac56-74b362e327c4_characterizeImage_855821_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/af2b664c-9f37-4363-aca5-6ee11507e9e9_isr_855821_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/af2b664c-9f37-4363-aca5-6ee11507e9e9_isr_855821_45.stderr
parsl.dataflow.dflow INFO: Task 684 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dbef9fc5591fc757f468c6d0de0c5a7e with result from task 684
parsl.dataflow.dflow DEBUG: Task 684 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 685 has memoization hash 18da18ee359a20d44e24baec39273936
parsl.dataflow.memoization INFO: Task 685 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15809620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 685 try 0 launched on executor multi with executor id 595
parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c351ec92-f25c-4c33-ac33-be5c8974635e_characterizeImage_855820_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c351ec92-f25c-4c33-ac33-be5c8974635e_characterizeImage_855820_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/f4a97481-681f-4dbd-b41a-9d278275fef9_isr_855820_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/f4a97481-681f-4dbd-b41a-9d278275fef9_isr_855820_7.stderr
parsl.dataflow.dflow INFO: Task 630 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4fa415be8fd027788e3f1a353aa5981c with result from task 630
parsl.dataflow.dflow DEBUG: Task 630 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 631 has memoization hash cf412a4688e2207429da7d5252ffdd5c
parsl.dataflow.memoization INFO: Task 631 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 631 try 0 launched on executor multi with executor id 596
parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c2ddb1fe-ab98-466b-9ddd-90bee6f60a42_characterizeImage_855820_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c2ddb1fe-ab98-466b-9ddd-90bee6f60a42_characterizeImage_855820_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/bd081662-815f-4abf-8863-24827cf0963e_isr_855820_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/bd081662-815f-4abf-8863-24827cf0963e_isr_855820_8.stderr
parsl.dataflow.dflow INFO: Task 612 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8fc0b462d75ec58ab057117b0a1b131a with result from task 612
parsl.dataflow.dflow DEBUG: Task 612 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 613 has memoization hash b8e6f36aa6b4b7d30e697ddad8b9c5f5
parsl.dataflow.memoization INFO: Task 613 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 613 try 0 launched on executor multi with executor id 597
parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9a44afea-f625-4145-af9b-9f4483525f5e_characterizeImage_855910_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9a44afea-f625-4145-af9b-9f4483525f5e_characterizeImage_855910_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/7c078c84-2c06-4ef5-9cc1-dcb0f4e44c89_isr_855910_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/7c078c84-2c06-4ef5-9cc1-dcb0f4e44c89_isr_855910_22.stderr
parsl.dataflow.dflow INFO: Task 591 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d63f59fe3e1420a5ecfe0ca9297ae462 with result from task 591
parsl.dataflow.dflow DEBUG: Task 591 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 592 has memoization hash e7c425ee38fca5db753e4508dd5bc960
parsl.dataflow.memoization INFO: Task 592 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 592 try 0 launched on executor multi with executor id 598
parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/b2445c1f-dd66-42b6-8aae-5ae3af029f68_characterizeImage_855820_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/b2445c1f-dd66-42b6-8aae-5ae3af029f68_characterizeImage_855820_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/f84fab63-a7da-4e22-89ab-8fbea42712e9_isr_855820_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/f84fab63-a7da-4e22-89ab-8fbea42712e9_isr_855820_9.stderr
parsl.dataflow.dflow INFO: Task 609 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 61c3980390d911c44c8cd8ba44f6b69f with result from task 609
parsl.dataflow.dflow DEBUG: Task 609 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 610 has memoization hash 9073f68d1ebf3d2a22109d22664fd6ce
parsl.dataflow.memoization INFO: Task 610 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 610 try 0 launched on executor multi with executor id 599
parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/5750f29e-c021-4aaa-ae5b-09af6e2f5058_characterizeImage_855820_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/5750f29e-c021-4aaa-ae5b-09af6e2f5058_characterizeImage_855820_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/b1d81041-aa2b-48ec-825b-e2898f29bcc2_isr_855820_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/b1d81041-aa2b-48ec-825b-e2898f29bcc2_isr_855820_20.stderr
parsl.dataflow.dflow INFO: Task 597 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 27c113e6ac354ab21056325123795cf5 with result from task 597
parsl.dataflow.dflow DEBUG: Task 597 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 598 has memoization hash 5da7418a1d3e39241a277cee563ba437
parsl.dataflow.memoization INFO: Task 598 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 598 try 0 launched on executor multi with executor id 600
parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/59309771-a509-4f3c-9362-62b9a2e241e7_characterizeImage_855821_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/59309771-a509-4f3c-9362-62b9a2e241e7_characterizeImage_855821_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/73c7702e-3d69-47d2-83b5-efd9ddce1d94_isr_855821_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/73c7702e-3d69-47d2-83b5-efd9ddce1d94_isr_855821_8.stderr
parsl.dataflow.dflow INFO: Task 693 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4e71f9517634b9cd001147ebcda1ef0b with result from task 693
parsl.dataflow.dflow DEBUG: Task 693 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 694 has memoization hash e028cd437b0013743831baa511937486
parsl.dataflow.memoization INFO: Task 694 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15809f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 694 try 0 launched on executor multi with executor id 601
parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/99231e1d-b415-4de5-8510-c098fb36585f_characterizeImage_855911_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/99231e1d-b415-4de5-8510-c098fb36585f_characterizeImage_855911_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/31152fe2-2e7d-46ad-8a97-1eecd170a7b4_isr_855911_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/31152fe2-2e7d-46ad-8a97-1eecd170a7b4_isr_855911_44.stderr
parsl.dataflow.dflow INFO: Task 687 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1c6ea178e21a6b8bea4eb92f04eb83cf with result from task 687
parsl.dataflow.dflow DEBUG: Task 687 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 688 has memoization hash bda2fecdb22c638f1f7552671986c399
parsl.dataflow.memoization INFO: Task 688 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15809940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 688 try 0 launched on executor multi with executor id 602
parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/4cc337c6-113a-473d-a40c-ad11dd05b56c_characterizeImage_855821_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/4cc337c6-113a-473d-a40c-ad11dd05b56c_characterizeImage_855821_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/328c9de3-5408-4ce7-83af-9ec14f4a25de_isr_855821_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/328c9de3-5408-4ce7-83af-9ec14f4a25de_isr_855821_41.stderr
parsl.dataflow.dflow INFO: Task 690 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b9a649f466c4a2b51bb3988d9790a7aa with result from task 690
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:01 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow DEBUG: Task 690 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 691 has memoization hash ed7a63e97f2608ca47b2bfec1d7f3687
parsl.dataflow.memoization INFO: Task 691 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15809c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 691 try 0 launched on executor multi with executor id 603
parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/73bb3146-8820-49c9-abff-b3f5384f1dcf_characterizeImage_855822_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/73bb3146-8820-49c9-abff-b3f5384f1dcf_characterizeImage_855822_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/dd73294b-184a-4946-b93b-be3fdc1a32bc_isr_855822_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/dd73294b-184a-4946-b93b-be3fdc1a32bc_isr_855822_50.stderr
parsl.dataflow.dflow INFO: Task 594 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 34565380a75b92fbe2ef0a2859703d17 with result from task 594
parsl.dataflow.dflow DEBUG: Task 594 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 595 has memoization hash 0214a4bc1c9ca198f836085b853b28b1
parsl.dataflow.memoization INFO: Task 595 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 595 try 0 launched on executor multi with executor id 604
parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/b3c2fb9b-8755-4044-ac83-c5c1eb493a72_characterizeImage_855821_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/b3c2fb9b-8755-4044-ac83-c5c1eb493a72_characterizeImage_855821_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/7fed1f33-2b17-48ea-b1ea-bd5421843cc9_isr_855821_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/7fed1f33-2b17-48ea-b1ea-bd5421843cc9_isr_855821_11.stderr
parsl.dataflow.dflow INFO: Task 633 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0d629e3ebd5d589e0cd2bad1745104d2 with result from task 633
parsl.dataflow.dflow DEBUG: Task 633 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 634 has memoization hash b1b059b8d83e096da5a6a5af2b3f73fa
parsl.dataflow.memoization INFO: Task 634 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 634 try 0 launched on executor multi with executor id 605
parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ad2d3277-f757-4b87-b4ba-9ac08ea5adfd_characterizeImage_855820_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ad2d3277-f757-4b87-b4ba-9ac08ea5adfd_characterizeImage_855820_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/daee3196-f336-4169-8bd6-4ac69dbcd0ea_isr_855820_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/daee3196-f336-4169-8bd6-4ac69dbcd0ea_isr_855820_19.stderr
parsl.dataflow.dflow INFO: Task 615 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f0334225bb989407d3e5c38fe719683a with result from task 615
parsl.dataflow.dflow DEBUG: Task 615 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 616 has memoization hash e96d371890aa491733302d21a1b67712
parsl.dataflow.memoization INFO: Task 616 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 616 try 0 launched on executor multi with executor id 606
parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/314d9a2e-afb2-44b7-aae0-d60cc3469b3a_characterizeImage_855911_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/314d9a2e-afb2-44b7-aae0-d60cc3469b3a_characterizeImage_855911_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/13757c90-0ea4-41fa-bbdb-33880cbbfc7b_isr_855911_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/13757c90-0ea4-41fa-bbdb-33880cbbfc7b_isr_855911_17.stderr
parsl.dataflow.dflow INFO: Task 672 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 872d0e32e983cc9b8b489722c033880c with result from task 672
parsl.dataflow.dflow DEBUG: Task 672 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 673 has memoization hash 41e5ee806d33b9dd2ae4a1aba279636d
parsl.dataflow.memoization INFO: Task 673 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158089a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 673 try 0 launched on executor multi with executor id 607
parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/f7f8474e-66ed-44b2-b7d2-0060ec832879_characterizeImage_855821_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/f7f8474e-66ed-44b2-b7d2-0060ec832879_characterizeImage_855821_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/487c3ba3-82a2-4164-9564-83657118b9c6_isr_855821_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/487c3ba3-82a2-4164-9564-83657118b9c6_isr_855821_49.stderr
parsl.dataflow.dflow INFO: Task 513 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c1b15705fcad9dae82f090137133e538 with result from task 513
parsl.dataflow.dflow DEBUG: Task 513 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 514 has memoization hash 805a8f49a657086a45be339e7a2f4841
parsl.dataflow.memoization INFO: Task 514 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15946340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 514 try 0 launched on executor multi with executor id 608
parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/4c8d2724-36b6-4eb4-90d1-22de9d9cfed3_characterizeImage_855910_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/4c8d2724-36b6-4eb4-90d1-22de9d9cfed3_characterizeImage_855910_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/45c65f4f-700f-408b-9a87-640faf1f8f54_isr_855910_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/45c65f4f-700f-408b-9a87-640faf1f8f54_isr_855910_11.stderr
parsl.dataflow.dflow INFO: Task 555 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aa40416ea33becf85bddefe70950fb76 with result from task 555
parsl.dataflow.dflow DEBUG: Task 555 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 556 has memoization hash 01719a839fe8bb776e852148015f69c3
parsl.dataflow.memoization INFO: Task 556 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 556 try 0 launched on executor multi with executor id 609
parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/1528f555-9382-4d5f-a6c3-7dc1ee12e688_characterizeImage_855822_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/1528f555-9382-4d5f-a6c3-7dc1ee12e688_characterizeImage_855822_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/4e8584de-01e9-4b6e-a961-77c618744885_isr_855822_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/4e8584de-01e9-4b6e-a961-77c618744885_isr_855822_35.stderr
parsl.dataflow.dflow INFO: Task 522 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 511b6ca10ca87529e78476d38b21cc34 with result from task 522
parsl.dataflow.dflow DEBUG: Task 522 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 523 has memoization hash 14da39d7ae7ca6ff37e8aa91e63e56fc
parsl.dataflow.memoization INFO: Task 523 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15946ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 523 try 0 launched on executor multi with executor id 610
parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/6704df77-76f2-4617-a4e9-a2c039cec75f_characterizeImage_855909_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/6704df77-76f2-4617-a4e9-a2c039cec75f_characterizeImage_855909_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/88bb420b-e30a-40d2-815f-78922d37dd11_isr_855909_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/88bb420b-e30a-40d2-815f-78922d37dd11_isr_855909_14.stderr
parsl.dataflow.dflow INFO: Task 519 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 650dc56c86d26aee674fe5896a85ecbd with result from task 519
parsl.dataflow.dflow DEBUG: Task 519 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 520 has memoization hash af4f7ecce07b87824f97859da083cecb
parsl.dataflow.memoization INFO: Task 520 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15946980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 520 try 0 launched on executor multi with executor id 611
parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/01c702af-c157-42c4-9e9d-62292b95238f_characterizeImage_855911_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/01c702af-c157-42c4-9e9d-62292b95238f_characterizeImage_855911_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/c82dd4ba-1217-424a-b0bd-1fcf079e013c_isr_855911_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/c82dd4ba-1217-424a-b0bd-1fcf079e013c_isr_855911_11.stderr
parsl.dataflow.dflow INFO: Task 531 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c54d192225fd5a42d783a6095c627eb8 with result from task 531
parsl.dataflow.dflow DEBUG: Task 531 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 532 has memoization hash 7ae61edb24360387beb85e59a58531cc
parsl.dataflow.memoization INFO: Task 532 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15947600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 532 try 0 launched on executor multi with executor id 612
parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b28ff9c2-f409-4140-8f81-68513963b66a_characterizeImage_855910_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b28ff9c2-f409-4140-8f81-68513963b66a_characterizeImage_855910_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/f2974119-4935-4c24-9d9e-870b546d0a76_isr_855910_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/f2974119-4935-4c24-9d9e-870b546d0a76_isr_855910_28.stderr
parsl.dataflow.dflow INFO: Task 537 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cd29d6d64f6974d867931e39e0fa0967 with result from task 537
parsl.dataflow.dflow DEBUG: Task 537 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 538 has memoization hash fa226f7c646149c0bf71633ea6dada25
parsl.dataflow.memoization INFO: Task 538 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15947c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 538 try 0 launched on executor multi with executor id 613
parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/4a1425ab-70e7-466e-95cd-29a7c8aae0b1_characterizeImage_855911_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/4a1425ab-70e7-466e-95cd-29a7c8aae0b1_characterizeImage_855911_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/e9decf17-764c-4501-8a27-d6ea7cf65b95_isr_855911_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/e9decf17-764c-4501-8a27-d6ea7cf65b95_isr_855911_51.stderr
parsl.dataflow.dflow INFO: Task 534 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 520a2c11f430c7bc446257ab977f9dea with result from task 534
parsl.dataflow.dflow DEBUG: Task 534 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 535 has memoization hash 75591ac7e47e88fd29051187027adb55
parsl.dataflow.memoization INFO: Task 535 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15947920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 535 try 0 launched on executor multi with executor id 614
parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/01c9bbcc-a864-411c-b656-c9ebf1f23fef_characterizeImage_855822_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/01c9bbcc-a864-411c-b656-c9ebf1f23fef_characterizeImage_855822_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/55ee9a7d-eceb-4066-b94a-19d15e9ba08e_isr_855822_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/55ee9a7d-eceb-4066-b94a-19d15e9ba08e_isr_855822_27.stderr
parsl.dataflow.dflow INFO: Task 720 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 795bae18b5ccbda5bddf19d4592a4427 with result from task 720
parsl.dataflow.dflow DEBUG: Task 720 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 721 has memoization hash 900a9cada33524e4e1d7252a074c1b34
parsl.dataflow.memoization INFO: Task 721 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 721 try 0 launched on executor multi with executor id 615
parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/cdb1cee3-7fa1-4ebc-bcf0-a9ef1ae1cfec_characterizeImage_855911_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/cdb1cee3-7fa1-4ebc-bcf0-a9ef1ae1cfec_characterizeImage_855911_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/0c350f46-9160-4d48-8514-feff4707ab35_isr_855911_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/0c350f46-9160-4d48-8514-feff4707ab35_isr_855911_13.stderr
parsl.dataflow.dflow INFO: Task 624 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1b77bcb0ece42063cb951ced51cd6b9b with result from task 624
parsl.dataflow.dflow DEBUG: Task 624 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 625 has memoization hash d4831cbd6bff973fed130fcc1075df38
parsl.dataflow.memoization INFO: Task 625 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 625 try 0 launched on executor multi with executor id 616
parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/7e2f60b3-bac3-495e-8273-0545814d8b80_characterizeImage_855820_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/7e2f60b3-bac3-495e-8273-0545814d8b80_characterizeImage_855820_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/2b38f3e2-f61d-4a3b-b9e5-1f8e2480b279_isr_855820_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/2b38f3e2-f61d-4a3b-b9e5-1f8e2480b279_isr_855820_28.stderr
parsl.dataflow.dflow INFO: Task 525 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bcfb52ee0c04ac10dc4cfa2a47bd207e with result from task 525
parsl.dataflow.dflow DEBUG: Task 525 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 526 has memoization hash c5b2d01d66b6d5b7e2d59414a9aed428
parsl.dataflow.memoization INFO: Task 526 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15946fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 526 try 0 launched on executor multi with executor id 617
parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/d365bfd2-cfd5-44fd-b3bf-2795d8640625_characterizeImage_855911_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/d365bfd2-cfd5-44fd-b3bf-2795d8640625_characterizeImage_855911_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/3535da8f-d667-43d7-8994-3f6dc5858503_isr_855911_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/3535da8f-d667-43d7-8994-3f6dc5858503_isr_855911_52.stderr
parsl.dataflow.dflow INFO: Task 588 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eb64482a76e1f4c4e9fbfaa9d9d00b48 with result from task 588
parsl.dataflow.dflow DEBUG: Task 588 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 589 has memoization hash 98eade9ac5578605139a9f6ded2dba22
parsl.dataflow.memoization INFO: Task 589 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 589 try 0 launched on executor multi with executor id 618
parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/dbcece99-9605-4365-9c86-3edcd40507ab_characterizeImage_855911_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/dbcece99-9605-4365-9c86-3edcd40507ab_characterizeImage_855911_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/5e383860-1e75-4fd6-ad44-eb03281c056b_isr_855911_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/5e383860-1e75-4fd6-ad44-eb03281c056b_isr_855911_3.stderr
parsl.dataflow.dflow INFO: Task 621 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6273cd6b33c48e3a942fc95963a6ba20 with result from task 621
parsl.dataflow.dflow DEBUG: Task 621 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 622 has memoization hash b0d4c0f24bae188e7ff352428ec7f5fa
parsl.dataflow.memoization INFO: Task 622 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 622 try 0 launched on executor multi with executor id 619
parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ef21c387-1bbf-4c6b-8687-c3d6db89bbb1_characterizeImage_855821_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ef21c387-1bbf-4c6b-8687-c3d6db89bbb1_characterizeImage_855821_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/1571e2dc-56ec-4200-bd88-e43249792453_isr_855821_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/1571e2dc-56ec-4200-bd88-e43249792453_isr_855821_57.stderr
parsl.dataflow.dflow INFO: Task 654 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 07b2f740b6dc567d0c93ee787298b93a with result from task 654
parsl.dataflow.dflow DEBUG: Task 654 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 655 has memoization hash 49b70cc98ca7adae1e4ae507022a2c14
parsl.dataflow.memoization INFO: Task 655 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 655 try 0 launched on executor multi with executor id 620
parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/28d7671b-a642-4aa7-a1af-0f61db4d1708_characterizeImage_855822_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/28d7671b-a642-4aa7-a1af-0f61db4d1708_characterizeImage_855822_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/9df2a49f-3bb9-400b-9b45-45603ca2b42c_isr_855822_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/9df2a49f-3bb9-400b-9b45-45603ca2b42c_isr_855822_37.stderr
parsl.dataflow.dflow INFO: Task 540 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1a9197bf5825eb4d21f3543017180217 with result from task 540
parsl.dataflow.dflow DEBUG: Task 540 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 541 has memoization hash 2afeeab34b1fa3be3465709379808987
parsl.dataflow.memoization INFO: Task 541 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15947f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 541 try 0 launched on executor multi with executor id 621
parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/2f717859-0942-4d79-b214-59d39385fa48_characterizeImage_855911_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/2f717859-0942-4d79-b214-59d39385fa48_characterizeImage_855911_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/7f8d73c9-5cff-45ab-b9c5-ef57b5e40676_isr_855911_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/7f8d73c9-5cff-45ab-b9c5-ef57b5e40676_isr_855911_53.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 331 try 0 failed
parsl.dataflow.dflow INFO: Task 331 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/43ac2e76-0ffa-420b-952a-048e40aa8647_characterizeImage_855911_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/43ac2e76-0ffa-420b-952a-048e40aa8647_characterizeImage_855911_23.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 31c98fd7bcb29d807bc43f5178c741c7
parsl.dataflow.memoization INFO: Task 331 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16aba3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 331 try 1 launched on executor multi with executor id 622
parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/43ac2e76-0ffa-420b-952a-048e40aa8647_characterizeImage_855911_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/43ac2e76-0ffa-420b-952a-048e40aa8647_characterizeImage_855911_23.stderr
parsl.dataflow.dflow INFO: Task 1032 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b8174973bdcaa987881bb1942a6de8a3 with result from task 1032
parsl.dataflow.dflow DEBUG: Task 1032 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1033 has memoization hash fd6e9c5e482e9cf9dce63e5e08a30764
parsl.dataflow.memoization INFO: Task 1033 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1033 try 0 launched on executor multi with executor id 623
parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/70ccf5df-2145-40f9-9777-fdb2aa7ed66f_characterizeImage_855822_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/70ccf5df-2145-40f9-9777-fdb2aa7ed66f_characterizeImage_855822_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/16dcc6f9-8e68-43b4-bfec-ad5f502989b9_isr_855822_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/16dcc6f9-8e68-43b4-bfec-ad5f502989b9_isr_855822_4.stderr
parsl.dataflow.dflow INFO: Task 1047 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 87c4d511913f354618d9b61eb001c037 with result from task 1047
parsl.dataflow.dflow DEBUG: Task 1047 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1048 has memoization hash d7a41152fa38e92ef0eed37d5f8443c8
parsl.dataflow.memoization INFO: Task 1048 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1048 try 0 launched on executor multi with executor id 624
parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/f2385b88-a091-40b0-a28f-63d340c9ad4c_characterizeImage_855821_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/f2385b88-a091-40b0-a28f-63d340c9ad4c_characterizeImage_855821_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/c4835a8b-9e85-4ae3-983a-815913e97830_isr_855821_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/c4835a8b-9e85-4ae3-983a-815913e97830_isr_855821_28.stderr
parsl.dataflow.dflow INFO: Task 735 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2a750e03cc6ea07ae9c80b266dae77c9 with result from task 735
parsl.dataflow.dflow DEBUG: Task 735 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 736 has memoization hash 4043bee07013c2fe97a08bc46fdb175e
parsl.dataflow.memoization INFO: Task 736 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15858b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 736 try 0 launched on executor multi with executor id 625
parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/6d174999-a3bd-451a-b80d-5f6e31b813ae_characterizeImage_855821_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/6d174999-a3bd-451a-b80d-5f6e31b813ae_characterizeImage_855821_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/1242aa15-e5c4-455a-8993-00b06d196ac7_isr_855821_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/1242aa15-e5c4-455a-8993-00b06d196ac7_isr_855821_27.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 364 try 0 failed
parsl.dataflow.dflow INFO: Task 364 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/0488fa05-2443-4276-8395-c08857e44fb6_characterizeImage_855911_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/0488fa05-2443-4276-8395-c08857e44fb6_characterizeImage_855911_16.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 364 has memoization hash f4c0a937b2c8a4a5092a6b0aebf15ce2
parsl.dataflow.memoization INFO: Task 364 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 364 try 1 launched on executor multi with executor id 626
parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/0488fa05-2443-4276-8395-c08857e44fb6_characterizeImage_855911_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/0488fa05-2443-4276-8395-c08857e44fb6_characterizeImage_855911_16.stderr
parsl.dataflow.dflow INFO: Task 708 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2295a722db5698bcca473cc9dea3a5a6 with result from task 708
parsl.dataflow.dflow DEBUG: Task 708 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 709 has memoization hash 37005d6a6d1cd9fb6effc35f1fc4054d
parsl.dataflow.memoization INFO: Task 709 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 709 try 0 launched on executor multi with executor id 627
parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/41f4abe5-116f-4461-bb35-e4782aed1809_characterizeImage_855910_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/41f4abe5-116f-4461-bb35-e4782aed1809_characterizeImage_855910_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/39240542-7563-4c3e-8888-3f266876b357_isr_855910_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/39240542-7563-4c3e-8888-3f266876b357_isr_855910_40.stderr
parsl.dataflow.dflow INFO: Task 1035 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 266ed8db758f64182ec9c8c4365dc8d4 with result from task 1035
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:02 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow DEBUG: Task 1035 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1036 has memoization hash c99569d9b742a0c986ea77c05574387f
parsl.dataflow.memoization INFO: Task 1036 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1036 try 0 launched on executor multi with executor id 628
parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/f2069b30-9019-4363-9b93-edf5c3ec0866_characterizeImage_855820_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/f2069b30-9019-4363-9b93-edf5c3ec0866_characterizeImage_855820_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/a9f463ee-efcc-4cae-a92b-49aa12d1a1c6_isr_855820_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/a9f463ee-efcc-4cae-a92b-49aa12d1a1c6_isr_855820_57.stderr
parsl.dataflow.dflow INFO: Task 528 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry da2e349f3017287064d5aeb02308268d with result from task 528
parsl.dataflow.dflow DEBUG: Task 528 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 529 has memoization hash 68c41bfbfcc03223c40b4a994bb675a2
parsl.dataflow.memoization INFO: Task 529 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159472e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 529 try 0 launched on executor multi with executor id 629
parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a2c394e6-14a1-464d-bcc5-4504bca96af0_characterizeImage_855910_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a2c394e6-14a1-464d-bcc5-4504bca96af0_characterizeImage_855910_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/6a5be03d-92e4-4514-aaeb-14cf4f806312_isr_855910_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/6a5be03d-92e4-4514-aaeb-14cf4f806312_isr_855910_36.stderr
parsl.dataflow.dflow INFO: Task 1029 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3b636fd7f2fd9ecec4b6000c3c8d2093 with result from task 1029
parsl.dataflow.dflow DEBUG: Task 1029 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1030 has memoization hash 09fca4ab4cfd5af6df93d128df70ebe1
parsl.dataflow.memoization INFO: Task 1030 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1030 try 0 launched on executor multi with executor id 630
parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/64067b22-a739-4683-989a-e74110f922a6_characterizeImage_855821_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/64067b22-a739-4683-989a-e74110f922a6_characterizeImage_855821_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/4f3c85f1-4cde-4734-bd56-3b902f9bd5a0_isr_855821_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/4f3c85f1-4cde-4734-bd56-3b902f9bd5a0_isr_855821_19.stderr
parsl.dataflow.dflow INFO: Task 747 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry af000421b25ab3cf5b6f5bc7f29debd0 with result from task 747
parsl.dataflow.dflow DEBUG: Task 747 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 748 has memoization hash 778fb66115ffc30802009627f4f4ac7b
parsl.dataflow.memoization INFO: Task 748 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15859800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 748 try 0 launched on executor multi with executor id 631
parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/6f155b08-1a87-4394-886d-d48ed41b770e_characterizeImage_855821_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/6f155b08-1a87-4394-886d-d48ed41b770e_characterizeImage_855821_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/69013455-02bc-4ff6-a1c7-cda91ad3bdd6_isr_855821_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/69013455-02bc-4ff6-a1c7-cda91ad3bdd6_isr_855821_22.stderr
parsl.dataflow.dflow INFO: Task 552 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fb95d813d3604d41fd91b59ef8638168 with result from task 552
parsl.dataflow.dflow DEBUG: Task 552 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 553 has memoization hash 59d9135ca65e6617b453930184cae62e
parsl.dataflow.memoization INFO: Task 553 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 553 try 0 launched on executor multi with executor id 632
parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/d25760e0-7889-4b59-891c-3f7b383a6659_characterizeImage_855909_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/d25760e0-7889-4b59-891c-3f7b383a6659_characterizeImage_855909_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/4f5d5307-0b0b-46cb-bb97-e2cfb929cd97_isr_855909_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/4f5d5307-0b0b-46cb-bb97-e2cfb929cd97_isr_855909_48.stderr
parsl.dataflow.dflow INFO: Task 726 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 218c0a3d1f7331fe926da744b9cfe89a with result from task 726
parsl.dataflow.dflow DEBUG: Task 726 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 727 has memoization hash 0ea51a7dafb5b59c24c2af9e303449ee
parsl.dataflow.memoization INFO: Task 727 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 727 try 0 launched on executor multi with executor id 633
parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/2920bbf3-d993-43d3-bd19-e7726bd85d13_characterizeImage_855822_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/2920bbf3-d993-43d3-bd19-e7726bd85d13_characterizeImage_855822_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/66aa9739-d896-4bf1-b7e9-8333bfa79df1_isr_855822_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/66aa9739-d896-4bf1-b7e9-8333bfa79df1_isr_855822_31.stderr
parsl.dataflow.dflow INFO: Task 738 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 37a5699334d9a1d0eb1b65293e2a0910 with result from task 738
parsl.dataflow.dflow DEBUG: Task 738 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 739 has memoization hash 8f73dfb490880c4e831d6027bddb77e9
parsl.dataflow.memoization INFO: Task 739 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15858ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 739 try 0 launched on executor multi with executor id 634
parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/13685c98-bb4d-4402-96ca-6ed4012d88c4_characterizeImage_855909_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/13685c98-bb4d-4402-96ca-6ed4012d88c4_characterizeImage_855909_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/45ec7539-3cc6-4c20-9887-31ce526aac89_isr_855909_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/45ec7539-3cc6-4c20-9887-31ce526aac89_isr_855909_61.stderr
parsl.dataflow.dflow INFO: Task 750 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d777f60b0f27777b2c61161f81281d29 with result from task 750
parsl.dataflow.dflow DEBUG: Task 750 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 751 has memoization hash 3014026d02ff6831ef5034427d216632
parsl.dataflow.memoization INFO: Task 751 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15859b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 751 try 0 launched on executor multi with executor id 635
parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/da0d1552-f18a-4d65-b727-c59d69ca334a_characterizeImage_855911_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/da0d1552-f18a-4d65-b727-c59d69ca334a_characterizeImage_855911_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/f2de0941-d11d-4dd2-a7f7-29a05f7cfed9_isr_855911_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/f2de0941-d11d-4dd2-a7f7-29a05f7cfed9_isr_855911_46.stderr
parsl.dataflow.dflow INFO: Task 1044 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 240fd618ca387d52dc99a8a9b78c8074 with result from task 1044
parsl.dataflow.dflow DEBUG: Task 1044 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1045 has memoization hash 1fb7e3d86cbe569b3408e48585cf6d33
parsl.dataflow.memoization INFO: Task 1045 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1045 try 0 launched on executor multi with executor id 636
parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/d3e6683a-abd7-429f-b11c-82eb23369091_characterizeImage_855820_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/d3e6683a-abd7-429f-b11c-82eb23369091_characterizeImage_855820_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/85cd8538-007e-4d21-b05b-75e5b1e7c63d_isr_855820_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/85cd8538-007e-4d21-b05b-75e5b1e7c63d_isr_855820_17.stderr
parsl.dataflow.dflow INFO: Task 711 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c246af6af7ef788aeda0e5a3f37b583b with result from task 711
parsl.dataflow.dflow DEBUG: Task 711 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 712 has memoization hash d953d888a7e2164ad5898ef37639131d
parsl.dataflow.memoization INFO: Task 712 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 712 try 0 launched on executor multi with executor id 637
parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b8d5e5fe-2daf-48fe-949a-bea601c3e3df_characterizeImage_855910_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b8d5e5fe-2daf-48fe-949a-bea601c3e3df_characterizeImage_855910_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/127ea4be-46a8-4527-88d5-8fed1d0af41c_isr_855910_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/127ea4be-46a8-4527-88d5-8fed1d0af41c_isr_855910_13.stderr
parsl.dataflow.dflow INFO: Task 717 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c020376f865a232d431c52d9f47dfc6e with result from task 717
parsl.dataflow.dflow DEBUG: Task 717 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 718 has memoization hash f077e85d55c4299c61411034090a781c
parsl.dataflow.memoization INFO: Task 718 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 718 try 0 launched on executor multi with executor id 638
parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ea0cf7a9-f65a-4fd2-b5fe-88cc7369b661_characterizeImage_855820_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ea0cf7a9-f65a-4fd2-b5fe-88cc7369b661_characterizeImage_855820_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/52ab0021-3fa0-4fa4-8c7e-c3f72e2c182d_isr_855820_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/52ab0021-3fa0-4fa4-8c7e-c3f72e2c182d_isr_855820_41.stderr
parsl.dataflow.dflow INFO: Task 741 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3a74096c9f7af438940c537c18f4f1ec with result from task 741
parsl.dataflow.dflow DEBUG: Task 741 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 742 has memoization hash cad572f80e7451643ccc9b10f6d4f330
parsl.dataflow.memoization INFO: Task 742 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158591c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 742 try 0 launched on executor multi with executor id 639
parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/ee15b11f-7e12-460c-b283-16aa1a19a90b_characterizeImage_855910_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/ee15b11f-7e12-460c-b283-16aa1a19a90b_characterizeImage_855910_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/84c5640b-7aa6-4199-a2dd-1aa3b03c68b3_isr_855910_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/84c5640b-7aa6-4199-a2dd-1aa3b03c68b3_isr_855910_3.stderr
parsl.dataflow.dflow INFO: Task 1041 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2f4a2d5055b46435cff8700cfdd2cbdb with result from task 1041
parsl.dataflow.dflow DEBUG: Task 1041 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1042 has memoization hash c383d405c4b25e05360df755ee7af784
parsl.dataflow.memoization INFO: Task 1042 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1042 try 0 launched on executor multi with executor id 640
parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/7570243c-6ca2-44db-8516-bd1db9b982f4_characterizeImage_855911_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/7570243c-6ca2-44db-8516-bd1db9b982f4_characterizeImage_855911_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/c0e73a22-8459-4a99-9a09-b91ee46a226e_isr_855911_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/c0e73a22-8459-4a99-9a09-b91ee46a226e_isr_855911_45.stderr
parsl.dataflow.dflow INFO: Task 627 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 00e33f9331a09d44ce5ffa9c875fff5d with result from task 627
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:02 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow DEBUG: Task 627 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 628 has memoization hash 4b13bbd1d5a8735f365532899c42f585
parsl.dataflow.memoization INFO: Task 628 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 628 try 0 launched on executor multi with executor id 641
parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/3e7db2ef-b4dc-443e-920d-a65e1bc68ec7_characterizeImage_855822_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/3e7db2ef-b4dc-443e-920d-a65e1bc68ec7_characterizeImage_855822_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/dee9523a-cb65-419c-88d5-cf697a914f4a_isr_855822_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/dee9523a-cb65-419c-88d5-cf697a914f4a_isr_855822_7.stderr
parsl.dataflow.dflow INFO: Task 714 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3818d870b24ce700ba9df21d23da87d3 with result from task 714
parsl.dataflow.dflow DEBUG: Task 714 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 715 has memoization hash d82d4163e31d875a77caa0b92c4ea184
parsl.dataflow.memoization INFO: Task 715 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 715 try 0 launched on executor multi with executor id 642
parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/5cd4c424-ca65-479e-a5ad-c688f93d3f65_characterizeImage_855821_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/5cd4c424-ca65-479e-a5ad-c688f93d3f65_characterizeImage_855821_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/36fd6a43-4c21-4633-b7e9-c357a4226bb8_isr_855821_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/36fd6a43-4c21-4633-b7e9-c357a4226bb8_isr_855821_14.stderr
parsl.dataflow.dflow INFO: Task 657 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dde366b1c2381119f7745fcf191df056 with result from task 657
parsl.dataflow.dflow DEBUG: Task 657 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 658 has memoization hash 4dc7df2b7c9a20a70c951e0f5bbfa9a8
parsl.dataflow.memoization INFO: Task 658 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 658 try 0 launched on executor multi with executor id 643
parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/db297898-790d-432e-8d2e-d55ac959754d_characterizeImage_855909_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/db297898-790d-432e-8d2e-d55ac959754d_characterizeImage_855909_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/66915e4b-d5b6-4d47-94b5-238f5c5f425b_isr_855909_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/66915e4b-d5b6-4d47-94b5-238f5c5f425b_isr_855909_52.stderr
parsl.dataflow.dflow INFO: Task 600 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 234fd90f956d2ebfceb00ce5d3664dad with result from task 600
parsl.dataflow.dflow DEBUG: Task 600 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 601 has memoization hash 799da3ccd41cbcec596990cb04a62a15
parsl.dataflow.memoization INFO: Task 601 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 601 try 0 launched on executor multi with executor id 644
parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/c39b822e-04b6-4d49-8069-a41aba7b6c90_characterizeImage_855909_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/c39b822e-04b6-4d49-8069-a41aba7b6c90_characterizeImage_855909_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/a8b197d4-cea4-482a-abf4-e331f426daca_isr_855909_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/a8b197d4-cea4-482a-abf4-e331f426daca_isr_855909_28.stderr
parsl.dataflow.dflow INFO: Task 723 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 86406b7c904ccfd303fd0df5d667c907 with result from task 723
parsl.dataflow.dflow DEBUG: Task 723 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 724 has memoization hash 539a1468b63de63caaefd742a0154733
parsl.dataflow.memoization INFO: Task 724 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 724 try 0 launched on executor multi with executor id 645
parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/86f749f5-3fbf-4fb8-be29-20a0377b7713_characterizeImage_855911_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/86f749f5-3fbf-4fb8-be29-20a0377b7713_characterizeImage_855911_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/02f66a2c-c6cd-408b-9739-7c0b039ad12b_isr_855911_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/02f66a2c-c6cd-408b-9739-7c0b039ad12b_isr_855911_6.stderr
parsl.dataflow.dflow INFO: Task 543 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 455fce76f3a61b396ad90aab73297cf2 with result from task 543
parsl.dataflow.dflow DEBUG: Task 543 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 544 has memoization hash a03b06f450d9f791f7429cd822e28a48
parsl.dataflow.memoization INFO: Task 544 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 544 try 0 launched on executor multi with executor id 646
parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/2cbf3111-8af8-48cc-b869-ddf933599d2e_characterizeImage_855822_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/2cbf3111-8af8-48cc-b869-ddf933599d2e_characterizeImage_855822_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/522b43d8-cbbd-4aa7-a1b9-40f1ea9236a7_isr_855822_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/522b43d8-cbbd-4aa7-a1b9-40f1ea9236a7_isr_855822_3.stderr
parsl.dataflow.dflow INFO: Task 549 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 237bddaa8110521bb234af855db74521 with result from task 549
parsl.dataflow.dflow DEBUG: Task 549 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 550 has memoization hash 14e467cc7bd7a88f2bb9cf0aadeafe07
parsl.dataflow.memoization INFO: Task 550 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 550 try 0 launched on executor multi with executor id 647
parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/2b52761e-9b99-45d6-8c83-57d3861e9de5_characterizeImage_855821_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/2b52761e-9b99-45d6-8c83-57d3861e9de5_characterizeImage_855821_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/ebe913ae-c022-48a1-8738-299200ad6ebb_isr_855821_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/ebe913ae-c022-48a1-8738-299200ad6ebb_isr_855821_44.stderr
parsl.dataflow.dflow INFO: Task 1038 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c196abc422c3bca46262ec1da744e390 with result from task 1038
parsl.dataflow.dflow DEBUG: Task 1038 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1039 has memoization hash ee1f3e88476314832ac87aa22baf5f99
parsl.dataflow.memoization INFO: Task 1039 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1039 try 0 launched on executor multi with executor id 648
parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/043c5258-3f02-4bcd-a957-9e064c89d9b5_characterizeImage_855820_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/043c5258-3f02-4bcd-a957-9e064c89d9b5_characterizeImage_855820_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0c9ef19c-5af6-4f0f-98e6-e686d51066e7_isr_855820_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0c9ef19c-5af6-4f0f-98e6-e686d51066e7_isr_855820_39.stderr
parsl.dataflow.dflow INFO: Task 1050 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1f88418623f704b3995087d634b9e605 with result from task 1050
parsl.dataflow.dflow DEBUG: Task 1050 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1051 has memoization hash f4f4ae53e691e288193be263e0160b5e
parsl.dataflow.memoization INFO: Task 1051 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1051 try 0 launched on executor multi with executor id 649
parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/2ac33e9b-d573-4950-96fe-3c84aca88ad2_characterizeImage_855820_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/2ac33e9b-d573-4950-96fe-3c84aca88ad2_characterizeImage_855820_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/f8d7cfed-b61d-40f1-ad5f-caa89d0c5fb7_isr_855820_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/f8d7cfed-b61d-40f1-ad5f-caa89d0c5fb7_isr_855820_53.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:03 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 606 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f812a861fbb459d25a60924d613eff33 with result from task 606
parsl.dataflow.dflow DEBUG: Task 606 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 607 has memoization hash 3f0e36882c759643db2edc6c4c899c66
parsl.dataflow.memoization INFO: Task 607 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 607 try 0 launched on executor multi with executor id 650
parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/80ee0abb-6edb-479d-b69a-000e10a4d80d_characterizeImage_855822_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/80ee0abb-6edb-479d-b69a-000e10a4d80d_characterizeImage_855822_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/523f7230-ab5b-40ce-853a-e07c7428fdc3_isr_855822_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/523f7230-ab5b-40ce-853a-e07c7428fdc3_isr_855822_55.stderr
parsl.dataflow.dflow INFO: Task 729 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4216fc2de651cdf40e925f0ddec90bfd with result from task 729
parsl.dataflow.dflow DEBUG: Task 729 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 730 has memoization hash 93a774ead62126e57411be69fd0b5948
parsl.dataflow.memoization INFO: Task 730 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15858540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 730 try 0 launched on executor multi with executor id 651
parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/1717e1ed-b56e-4a49-a425-b54344460d32_characterizeImage_855911_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/1717e1ed-b56e-4a49-a425-b54344460d32_characterizeImage_855911_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/e2e04629-acba-4e6c-b5cf-3c29f43909fc_isr_855911_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/e2e04629-acba-4e6c-b5cf-3c29f43909fc_isr_855911_26.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 67 try 0 failed
parsl.dataflow.dflow INFO: Task 67 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/e48d4589-e0c5-4812-8dcb-94e46c3bd2fa_characterizeImage_855909_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/e48d4589-e0c5-4812-8dcb-94e46c3bd2fa_characterizeImage_855909_2.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 72dafb5d26fd924d15ba7296be976376
parsl.dataflow.memoization INFO: Task 67 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 67 try 1 launched on executor multi with executor id 652
parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/e48d4589-e0c5-4812-8dcb-94e46c3bd2fa_characterizeImage_855909_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/e48d4589-e0c5-4812-8dcb-94e46c3bd2fa_characterizeImage_855909_2.stderr
parsl.dataflow.dflow INFO: Task 546 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 176f277f98ecd94da17549b174e455e6 with result from task 546
parsl.dataflow.dflow DEBUG: Task 546 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 547 has memoization hash a24e55ebc0baf166d185a590027f488a
parsl.dataflow.memoization INFO: Task 547 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 547 try 0 launched on executor multi with executor id 653
parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/983ad20f-c34f-4fc4-9da3-a6ef678a1ddb_characterizeImage_855910_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/983ad20f-c34f-4fc4-9da3-a6ef678a1ddb_characterizeImage_855910_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/ad51fae2-0fe9-47b9-b0e3-ed899f41ae08_isr_855910_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/ad51fae2-0fe9-47b9-b0e3-ed899f41ae08_isr_855910_62.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 793 try 0 failed
parsl.dataflow.dflow INFO: Task 793 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/3c16f8fe-60c8-44fc-979e-75cc5f8ebe10_characterizeImage_855911_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/3c16f8fe-60c8-44fc-979e-75cc5f8ebe10_characterizeImage_855911_40.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 793 has memoization hash ab743d0a9411a9bc83138a49c9fe9623
parsl.dataflow.memoization INFO: Task 793 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15894720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 793 try 1 launched on executor multi with executor id 654
parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/3c16f8fe-60c8-44fc-979e-75cc5f8ebe10_characterizeImage_855911_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/3c16f8fe-60c8-44fc-979e-75cc5f8ebe10_characterizeImage_855911_40.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 76 try 0 failed
parsl.dataflow.dflow INFO: Task 76 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a7ba73de-60e6-4bf8-a7dd-e884c059c8dd_characterizeImage_855822_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a7ba73de-60e6-4bf8-a7dd-e884c059c8dd_characterizeImage_855822_2.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 6d86c1738ee5ffe58ba6fffb442a5fea
parsl.dataflow.memoization INFO: Task 76 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 76 try 1 launched on executor multi with executor id 655
parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a7ba73de-60e6-4bf8-a7dd-e884c059c8dd_characterizeImage_855822_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a7ba73de-60e6-4bf8-a7dd-e884c059c8dd_characterizeImage_855822_2.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:04 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 558 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b100b6f5f65fcbe3c665dd4882a18557 with result from task 558
parsl.dataflow.dflow DEBUG: Task 558 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 559 has memoization hash 370fdfc9d61c5013360ffd6c9761c36c
parsl.dataflow.memoization INFO: Task 559 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 559 try 0 launched on executor multi with executor id 656
parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/efaa552c-ea8a-4119-aaaf-07272407d8f1_characterizeImage_855909_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/efaa552c-ea8a-4119-aaaf-07272407d8f1_characterizeImage_855909_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/ebaa991c-af56-4a4e-97a0-9f056406e160_isr_855909_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/ebaa991c-af56-4a4e-97a0-9f056406e160_isr_855909_53.stderr
parsl.dataflow.dflow INFO: Task 618 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ccf968ea855557950f7255fc9141c661 with result from task 618
parsl.dataflow.dflow DEBUG: Task 618 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 619 has memoization hash 1a18198a4f4c3a99bf02b7fec85b3de8
parsl.dataflow.memoization INFO: Task 619 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 619 try 0 launched on executor multi with executor id 657
parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/1e8ac2dc-3a26-4bea-9767-d7aa40c2b94b_characterizeImage_855909_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/1e8ac2dc-3a26-4bea-9767-d7aa40c2b94b_characterizeImage_855909_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/42f79313-b6fd-4f99-b911-a98977397f7c_isr_855909_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/42f79313-b6fd-4f99-b911-a98977397f7c_isr_855909_54.stderr
parsl.dataflow.dflow INFO: Task 753 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bcd31f770cf139c618e4eb05722843d2 with result from task 753
parsl.dataflow.dflow DEBUG: Task 753 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 754 has memoization hash e731a5119abc57c7bb562192418fab12
parsl.dataflow.memoization INFO: Task 754 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15859e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 754 try 0 launched on executor multi with executor id 658
parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/7796db12-2c74-47de-ab9b-64c7763d13a9_characterizeImage_855822_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/7796db12-2c74-47de-ab9b-64c7763d13a9_characterizeImage_855822_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/b09f437c-d848-4f92-8aa5-1da8a13470ef_isr_855822_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/b09f437c-d848-4f92-8aa5-1da8a13470ef_isr_855822_47.stderr
parsl.dataflow.dflow INFO: Task 603 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8fbdf14491c64bef3191fef37dfffd15 with result from task 603
parsl.dataflow.dflow DEBUG: Task 603 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 604 has memoization hash 0dd5d1269c52769f4ab68dd7586ab31c
parsl.dataflow.memoization INFO: Task 604 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 604 try 0 launched on executor multi with executor id 659
parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/70ee040c-fc43-43b5-8d66-1315f5be4080_characterizeImage_855822_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/70ee040c-fc43-43b5-8d66-1315f5be4080_characterizeImage_855822_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/bee74196-931d-4e05-8aae-19bee9c15bee_isr_855822_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/bee74196-931d-4e05-8aae-19bee9c15bee_isr_855822_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:04 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 756 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c0fa94ba829e9c77388e6af973adbf49 with result from task 756
parsl.dataflow.dflow DEBUG: Task 756 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 757 has memoization hash f31c90d08c8b7a684d18e925f1084b88
parsl.dataflow.memoization INFO: Task 757 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 757 try 0 launched on executor multi with executor id 660
parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/1b0fa141-3af7-4616-937f-9bafc2349196_characterizeImage_855910_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/1b0fa141-3af7-4616-937f-9bafc2349196_characterizeImage_855910_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/b3606f6b-4001-4028-85c7-7c58e468c43f_isr_855910_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/b3606f6b-4001-4028-85c7-7c58e468c43f_isr_855910_12.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 4 try 0 failed
parsl.dataflow.dflow INFO: Task 4 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/3876bf01-95f0-4738-8e5d-a3052cf869b8_characterizeImage_855911_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/3876bf01-95f0-4738-8e5d-a3052cf869b8_characterizeImage_855911_33.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 9eec1aeaf65e20888775993fac73ff1f
parsl.dataflow.memoization INFO: Task 4 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fcb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 4 try 1 launched on executor multi with executor id 661
parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/3876bf01-95f0-4738-8e5d-a3052cf869b8_characterizeImage_855911_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/3876bf01-95f0-4738-8e5d-a3052cf869b8_characterizeImage_855911_33.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 346 try 0 failed
parsl.dataflow.dflow INFO: Task 346 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7e0bf724-a995-4352-b9f6-334d12aed276_characterizeImage_855910_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7e0bf724-a995-4352-b9f6-334d12aed276_characterizeImage_855910_27.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 346 has memoization hash 0c1a81737dba7926fed2c9acca6004e9
parsl.dataflow.memoization INFO: Task 346 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16abb380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 346 try 1 launched on executor multi with executor id 662
parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7e0bf724-a995-4352-b9f6-334d12aed276_characterizeImage_855910_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7e0bf724-a995-4352-b9f6-334d12aed276_characterizeImage_855910_27.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 371
parsl.dataflow.strategy DEBUG: Executor multi has 371 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 20:56:05 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
parsl.dataflow.dflow INFO: Task 516 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 848f14e1cf13e4e412c786921138bdcb with result from task 516
parsl.dataflow.dflow DEBUG: Task 516 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 517 has memoization hash ede42ae69dafb537e02179e558af1275
parsl.dataflow.memoization INFO: Task 517 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15946660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 517 try 0 launched on executor multi with executor id 663
parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/cd8000da-0d9d-4870-9e74-39cb1774ecf2_characterizeImage_855820_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/cd8000da-0d9d-4870-9e74-39cb1774ecf2_characterizeImage_855820_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/3f1b5134-b85a-412f-8491-80e7d6cfb238_isr_855820_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/3f1b5134-b85a-412f-8491-80e7d6cfb238_isr_855820_38.stderr
2024-03-09 20:56:05 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9d66b7a2145248f52c1f9ae36e58215c with result from task 136
parsl.dataflow.dflow DEBUG: Task 136 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 89b171c4fe9324fbb932a23dd3990938
parsl.dataflow.memoization INFO: Task 137 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a41620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 664
parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/2a3bf82d-e168-4b5f-a130-e8623b2cfef7_calibrate_855910_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/2a3bf82d-e168-4b5f-a130-e8623b2cfef7_calibrate_855910_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/91bbf13d-1849-42e0-984a-e0e2b17d282e_characterizeImage_855910_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/91bbf13d-1849-42e0-984a-e0e2b17d282e_characterizeImage_855910_4.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:06 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:06 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 1113 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cc70999d306a4d9349f74e3cba9dfd94 with result from task 1113
parsl.dataflow.dflow DEBUG: Task 1113 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1114 has memoization hash 1de023253abcf145d857ef1bdc44cd3b
parsl.dataflow.memoization INFO: Task 1114 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1560d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1114 try 0 launched on executor multi with executor id 665
parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/2650a97b-d793-4f1a-a4f8-d3640c0d5e6d_characterizeImage_855822_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/2650a97b-d793-4f1a-a4f8-d3640c0d5e6d_characterizeImage_855822_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/05d4f8da-75dd-49e6-b3a9-e0337c23d0c1_isr_855822_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/05d4f8da-75dd-49e6-b3a9-e0337c23d0c1_isr_855822_14.stderr
parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f2f4e64e8d0b3ae7a7d5197758fe2bce with result from task 97
parsl.dataflow.dflow DEBUG: Task 97 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 98 has memoization hash f53c12806536d1a47d8fcee38131e21b
parsl.dataflow.memoization INFO: Task 98 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 666
parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/7f728b61-7bd1-447a-a446-2b59f819f43f_calibrate_855911_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/7f728b61-7bd1-447a-a446-2b59f819f43f_calibrate_855911_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/b781e9c5-cb24-41ab-abf7-9d9ed9968d05_characterizeImage_855911_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/b781e9c5-cb24-41ab-abf7-9d9ed9968d05_characterizeImage_855911_55.stderr
parsl.dataflow.dflow INFO: Task 855 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 53fd3ff028234be2889ca1895eceddfb with result from task 855
parsl.dataflow.dflow DEBUG: Task 855 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 856 has memoization hash 50323c4d45677dfd476c835927b8c49f
parsl.dataflow.memoization INFO: Task 856 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 856 try 0 launched on executor multi with executor id 667
parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/7eb975e2-6856-4b04-9835-51b298f27403_characterizeImage_855820_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/7eb975e2-6856-4b04-9835-51b298f27403_characterizeImage_855820_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0e781439-cc63-4721-a37c-d401c2f90534_isr_855820_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0e781439-cc63-4721-a37c-d401c2f90534_isr_855820_35.stderr
parsl.dataflow.dflow INFO: Task 852 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 161aba1f72a9e296ad322ae644c275db with result from task 852
parsl.dataflow.dflow DEBUG: Task 852 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 853 has memoization hash 36533102723a59e12e32cb8262064078
parsl.dataflow.memoization INFO: Task 853 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 853 try 0 launched on executor multi with executor id 668
parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/04860449-e628-494c-b38f-a95dcfbcd802_characterizeImage_855909_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/04860449-e628-494c-b38f-a95dcfbcd802_characterizeImage_855909_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/1361dd39-4a1f-4a6a-88dc-2c0a209f0aeb_isr_855909_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/1361dd39-4a1f-4a6a-88dc-2c0a209f0aeb_isr_855909_62.stderr
parsl.dataflow.dflow INFO: Task 864 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 452ba0649c1a92593f114eb4d879b693 with result from task 864
parsl.dataflow.dflow DEBUG: Task 864 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 865 has memoization hash 686bb50b6884b54c4728ea4bf3cc6d73
parsl.dataflow.memoization INFO: Task 865 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 865 try 0 launched on executor multi with executor id 669
parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b1ec9807-3b0b-477f-ab45-6e7e6ba3d354_characterizeImage_855910_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b1ec9807-3b0b-477f-ab45-6e7e6ba3d354_characterizeImage_855910_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/e511decf-a3d7-4477-9622-94d91e22ba5c_isr_855910_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/e511decf-a3d7-4477-9622-94d91e22ba5c_isr_855910_35.stderr
parsl.dataflow.dflow INFO: Task 849 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 227ccf6f7f88dc3a80845d81a7c512d9 with result from task 849
parsl.dataflow.dflow DEBUG: Task 849 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 850 has memoization hash bf770d3689c08d0791ad7139e4c29c49
parsl.dataflow.memoization INFO: Task 850 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 850 try 0 launched on executor multi with executor id 670
parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/11a08ca0-e0d9-4786-9d91-19aed048bf37_characterizeImage_855909_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/11a08ca0-e0d9-4786-9d91-19aed048bf37_characterizeImage_855909_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/31f31ed7-8c9d-463d-afda-a8ececb14448_isr_855909_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/31f31ed7-8c9d-463d-afda-a8ececb14448_isr_855909_13.stderr
parsl.dataflow.dflow INFO: Task 846 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7c307c55b37a95d290249555c06684b7 with result from task 846
parsl.dataflow.dflow DEBUG: Task 846 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 847 has memoization hash 8242a98cac56c33208355dd500abf3df
parsl.dataflow.memoization INFO: Task 847 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15897f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 847 try 0 launched on executor multi with executor id 671
parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/f85e8d7d-c248-499a-aa41-99ed2a946c91_characterizeImage_855910_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/f85e8d7d-c248-499a-aa41-99ed2a946c91_characterizeImage_855910_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/298e680c-2fe8-4dd6-8210-9ac53ec18774_isr_855910_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/298e680c-2fe8-4dd6-8210-9ac53ec18774_isr_855910_5.stderr
parsl.dataflow.dflow INFO: Task 861 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f008c05cc2a84430d576180e167bb3b5 with result from task 861
parsl.dataflow.dflow DEBUG: Task 861 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 862 has memoization hash ab2e436003bc574cc0c85e4995dda5c2
parsl.dataflow.memoization INFO: Task 862 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 862 try 0 launched on executor multi with executor id 672
parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/268f5621-29f5-4259-963d-55edc671e1df_characterizeImage_855821_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/268f5621-29f5-4259-963d-55edc671e1df_characterizeImage_855821_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/d6c6aa68-0427-4194-8fbe-16ab7173c96f_isr_855821_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/d6c6aa68-0427-4194-8fbe-16ab7173c96f_isr_855821_53.stderr
parsl.dataflow.dflow INFO: Task 858 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2f7027430a6c36dc4bc4ea1cd2e22f66 with result from task 858
parsl.dataflow.dflow DEBUG: Task 858 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 859 has memoization hash 52479442d118ec5e09fbd1e759df2106
parsl.dataflow.memoization INFO: Task 859 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 859 try 0 launched on executor multi with executor id 673
parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/6e2e5b54-4452-46b0-a1e7-b04edf11655a_characterizeImage_855909_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/6e2e5b54-4452-46b0-a1e7-b04edf11655a_characterizeImage_855909_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/47e7974e-c8a0-418a-9029-0278f25ed9de_isr_855909_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/47e7974e-c8a0-418a-9029-0278f25ed9de_isr_855909_35.stderr
parsl.dataflow.dflow INFO: Task 867 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 23c34c11b27ca532ad83d348afe1187a with result from task 867
parsl.dataflow.dflow DEBUG: Task 867 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 868 has memoization hash 81ac97f76a158840d137a04fc95ca40b
parsl.dataflow.memoization INFO: Task 868 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 868 try 0 launched on executor multi with executor id 674
parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/6ca1be66-963a-4a4c-b7f1-d1916172e9aa_characterizeImage_855822_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/6ca1be66-963a-4a4c-b7f1-d1916172e9aa_characterizeImage_855822_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/1f915565-a8a1-42ae-8fb5-262cdfbdc8d7_isr_855822_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/1f915565-a8a1-42ae-8fb5-262cdfbdc8d7_isr_855822_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:06 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 1110 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4f1c424b4bac8838728fe2d029a887e3 with result from task 1110
parsl.dataflow.dflow DEBUG: Task 1110 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1111 has memoization hash 669c0ba0f2011f5b009b04ac1aa81983
parsl.dataflow.memoization INFO: Task 1111 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1560d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1111 try 0 launched on executor multi with executor id 675
parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/8b0b0ebc-5adf-4041-9f8a-12a766517039_characterizeImage_855910_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/8b0b0ebc-5adf-4041-9f8a-12a766517039_characterizeImage_855910_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/b22dae0c-c28c-43e3-8921-9aa1bc8fd8ab_isr_855910_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/b22dae0c-c28c-43e3-8921-9aa1bc8fd8ab_isr_855910_24.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:07 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 1107 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c6489b52da513eac6575a7e8d35c5474 with result from task 1107
parsl.dataflow.dflow DEBUG: Task 1107 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1108 has memoization hash beedc52c983e98c044d80cbe14323efd
parsl.dataflow.memoization INFO: Task 1108 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1560d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1108 try 0 launched on executor multi with executor id 676
parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/cf847fe0-7e23-4dbc-883d-794cea1a8aec_characterizeImage_855911_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/cf847fe0-7e23-4dbc-883d-794cea1a8aec_characterizeImage_855911_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/1e844243-0f43-41df-85ca-ab0b54c618e8_isr_855911_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/1e844243-0f43-41df-85ca-ab0b54c618e8_isr_855911_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:07 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 780 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8938f3ca7c56d6a4e6b1211c5fe98194 with result from task 780
parsl.dataflow.dflow DEBUG: Task 780 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 781 has memoization hash 587174573fc0877ed7a61c0e8a10e16a
parsl.dataflow.memoization INFO: Task 781 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 781 try 0 launched on executor multi with executor id 677
parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/f63b146a-ab5d-4c8e-b646-bf1e3a8ab01e_characterizeImage_855911_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/f63b146a-ab5d-4c8e-b646-bf1e3a8ab01e_characterizeImage_855911_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/f264fae1-3a01-43a0-b6b7-20c31e05c08e_isr_855911_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/f264fae1-3a01-43a0-b6b7-20c31e05c08e_isr_855911_24.stderr
parsl.dataflow.dflow INFO: Task 783 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 63449c431d7ef07a6786c7fd98250efc with result from task 783
parsl.dataflow.dflow DEBUG: Task 783 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 784 has memoization hash 3e4782217010717f700cd3bd61343a43
parsl.dataflow.memoization INFO: Task 784 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 784 try 0 launched on executor multi with executor id 678
parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/dd7f1490-10b2-44c5-a417-10c061b85a61_characterizeImage_855821_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/dd7f1490-10b2-44c5-a417-10c061b85a61_characterizeImage_855821_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/c9689da1-0737-45f0-a1eb-508acbae4e1d_isr_855821_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/c9689da1-0737-45f0-a1eb-508acbae4e1d_isr_855821_23.stderr
parsl.dataflow.dflow INFO: Task 774 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 99f22b5276dacaefeaffbf55dd3987dd with result from task 774
parsl.dataflow.dflow DEBUG: Task 774 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 775 has memoization hash cf5a7c90680a16a91f7cb0fd261fd308
parsl.dataflow.memoization INFO: Task 775 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 775 try 0 launched on executor multi with executor id 679
parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/e98d6a3d-3cb7-4a89-a258-3c13a73e7e52_characterizeImage_855822_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/e98d6a3d-3cb7-4a89-a258-3c13a73e7e52_characterizeImage_855822_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/88dab2ac-c3c1-40b0-8adf-1f63c055c027_isr_855822_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/88dab2ac-c3c1-40b0-8adf-1f63c055c027_isr_855822_26.stderr
parsl.dataflow.dflow INFO: Task 777 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8f6b0838b99f8bf1b52897f551e7f125 with result from task 777
parsl.dataflow.dflow DEBUG: Task 777 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 778 has memoization hash e7716551a4f93a433da0d22a36b9737e
parsl.dataflow.memoization INFO: Task 778 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 778 try 0 launched on executor multi with executor id 680
parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/b9d511e0-45af-4f43-a4d1-b9b27bba55fc_characterizeImage_855909_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/b9d511e0-45af-4f43-a4d1-b9b27bba55fc_characterizeImage_855909_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/d765b95b-58af-4a43-bfea-95d99333737c_isr_855909_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/d765b95b-58af-4a43-bfea-95d99333737c_isr_855909_8.stderr
parsl.dataflow.dflow INFO: Task 768 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 220e8a179aa28e7221683b7904412504 with result from task 768
parsl.dataflow.dflow DEBUG: Task 768 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 769 has memoization hash 8c860e386ac2cd8013616714e50e0dd6
parsl.dataflow.memoization INFO: Task 769 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 769 try 0 launched on executor multi with executor id 681
parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/c3b10488-92b9-4c2f-bac8-a7082a9178d2_characterizeImage_855821_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/c3b10488-92b9-4c2f-bac8-a7082a9178d2_characterizeImage_855821_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/22173b18-8790-4209-8d6c-3b8dfd695d74_isr_855821_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/22173b18-8790-4209-8d6c-3b8dfd695d74_isr_855821_43.stderr
parsl.dataflow.dflow INFO: Task 771 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 93616cefb1503a7a636805517a81237a with result from task 771
parsl.dataflow.dflow DEBUG: Task 771 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 772 has memoization hash f3425e06005113529c06063cee997bf5
parsl.dataflow.memoization INFO: Task 772 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 772 try 0 launched on executor multi with executor id 682
parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/9ca65032-6e57-48cd-9c98-5a816016a451_characterizeImage_855820_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/9ca65032-6e57-48cd-9c98-5a816016a451_characterizeImage_855820_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/74bf6a4e-f186-47f4-be21-abb6012e4609_isr_855820_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/74bf6a4e-f186-47f4-be21-abb6012e4609_isr_855820_31.stderr
parsl.dataflow.dflow INFO: Task 786 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a69df3ba58fbd261b9ddee2ddaa521b2 with result from task 786
parsl.dataflow.dflow DEBUG: Task 786 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 787 has memoization hash 983920ad48c6e8322f61365585e9ccc4
parsl.dataflow.memoization INFO: Task 787 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 787 try 0 launched on executor multi with executor id 683
parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/adf5c93a-f1f7-4c1f-a9c6-5cdc7acb34fa_characterizeImage_855821_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/adf5c93a-f1f7-4c1f-a9c6-5cdc7acb34fa_characterizeImage_855821_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/257922e6-6174-4f1a-8326-33f93668e4fd_isr_855821_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/257922e6-6174-4f1a-8326-33f93668e4fd_isr_855821_29.stderr
parsl.dataflow.dflow INFO: Task 765 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cc8e22486c46a81bb2dc63de2ed7bb3e with result from task 765
parsl.dataflow.dflow DEBUG: Task 765 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 766 has memoization hash a18a0f97000fb9e83c636b566aa1f9d9
parsl.dataflow.memoization INFO: Task 766 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 766 try 0 launched on executor multi with executor id 684
parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/97fe3d8b-e53d-42d2-85b1-2ad2d8005928_characterizeImage_855910_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/97fe3d8b-e53d-42d2-85b1-2ad2d8005928_characterizeImage_855910_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/d9ebb859-4b60-4478-8874-3d8719c239cd_isr_855910_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/d9ebb859-4b60-4478-8874-3d8719c239cd_isr_855910_59.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 430 try 0 failed
parsl.dataflow.dflow INFO: Task 430 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7257da59-452a-496b-95ff-a7fe3d3cef53_characterizeImage_855910_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7257da59-452a-496b-95ff-a7fe3d3cef53_characterizeImage_855910_2.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 430 has memoization hash 3b27aea4bd6c4c814410fb1834776b53
parsl.dataflow.memoization INFO: Task 430 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 430 try 1 launched on executor multi with executor id 685
parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7257da59-452a-496b-95ff-a7fe3d3cef53_characterizeImage_855910_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7257da59-452a-496b-95ff-a7fe3d3cef53_characterizeImage_855910_2.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 313 try 0 failed
parsl.dataflow.dflow INFO: Task 313 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8f7cdbb3-2180-4a76-a27b-d6bf52a1eebd_characterizeImage_855911_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8f7cdbb3-2180-4a76-a27b-d6bf52a1eebd_characterizeImage_855911_15.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 313 has memoization hash a73354e138dcf9639491de69a740b98c
parsl.dataflow.memoization INFO: Task 313 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 313 try 1 launched on executor multi with executor id 686
parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8f7cdbb3-2180-4a76-a27b-d6bf52a1eebd_characterizeImage_855911_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8f7cdbb3-2180-4a76-a27b-d6bf52a1eebd_characterizeImage_855911_15.stderr
parsl.dataflow.dflow INFO: Task 801 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e2cc255f7cb3512da79041b1f4ec486e with result from task 801
parsl.dataflow.dflow DEBUG: Task 801 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 802 has memoization hash 31c89347931d428ba7178e70dbd3a85a
parsl.dataflow.memoization INFO: Task 802 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15895080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 802 try 0 launched on executor multi with executor id 687
parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a3f165c3-e409-4667-a596-123d9ae5bbee_characterizeImage_855910_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a3f165c3-e409-4667-a596-123d9ae5bbee_characterizeImage_855910_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/93ed9b76-d6e9-4892-a484-65151bab292e_isr_855910_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/93ed9b76-d6e9-4892-a484-65151bab292e_isr_855910_39.stderr
parsl.dataflow.dflow INFO: Task 909 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6dfbcf76c03105e72af5647c401a04a2 with result from task 909
parsl.dataflow.dflow DEBUG: Task 909 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 910 has memoization hash f21c3abfdaf034cda0458bb7fd4105d0
parsl.dataflow.memoization INFO: Task 910 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156ebf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 910 try 0 launched on executor multi with executor id 688
parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/9373a00d-22d0-4fc1-88f7-c296df4c2e2b_characterizeImage_855820_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/9373a00d-22d0-4fc1-88f7-c296df4c2e2b_characterizeImage_855820_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0c847ae1-3ae3-44f0-a563-0c12e033b723_isr_855820_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0c847ae1-3ae3-44f0-a563-0c12e033b723_isr_855820_34.stderr
parsl.dataflow.dflow INFO: Task 900 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 18697ba6752754e09ddc11e6aa9a3e48 with result from task 900
parsl.dataflow.dflow DEBUG: Task 900 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 901 has memoization hash 11f5102fdeb90e7c6f8b23e51c3af710
parsl.dataflow.memoization INFO: Task 901 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156eb7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 901 try 0 launched on executor multi with executor id 689
parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/0002dd9d-719a-4f3a-a3c4-5ad2824dc8e6_characterizeImage_855821_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/0002dd9d-719a-4f3a-a3c4-5ad2824dc8e6_characterizeImage_855821_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/ef0fc65c-19eb-4763-bec2-edd2c9bdf346_isr_855821_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/ef0fc65c-19eb-4763-bec2-edd2c9bdf346_isr_855821_39.stderr
parsl.dataflow.dflow INFO: Task 915 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fdee21e8d9eb42edea28612b3b193fc1 with result from task 915
parsl.dataflow.dflow DEBUG: Task 915 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 916 has memoization hash 78e53eac52ef3737f534ad136489cd4e
parsl.dataflow.memoization INFO: Task 916 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157287c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 916 try 0 launched on executor multi with executor id 690
parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/0bdf3ccb-bd3f-403f-bbec-3370ee7bb05f_characterizeImage_855910_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/0bdf3ccb-bd3f-403f-bbec-3370ee7bb05f_characterizeImage_855910_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/09d543f0-8abf-497b-8e0c-050b9b945ace_isr_855910_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/09d543f0-8abf-497b-8e0c-050b9b945ace_isr_855910_6.stderr
parsl.dataflow.dflow INFO: Task 912 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f98c71705f0112753c35c17f6ee8a55c with result from task 912
parsl.dataflow.dflow DEBUG: Task 912 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 913 has memoization hash 0019dd4534ec95365f8fdd85ca1abd1b
parsl.dataflow.memoization INFO: Task 913 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157284a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 913 try 0 launched on executor multi with executor id 691
parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/cb8d9282-f6ee-47b2-b580-a53006f2c1ce_characterizeImage_855910_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/cb8d9282-f6ee-47b2-b580-a53006f2c1ce_characterizeImage_855910_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/461db0e7-15e7-46fc-aeb4-c828aef366b6_isr_855910_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/461db0e7-15e7-46fc-aeb4-c828aef366b6_isr_855910_51.stderr
parsl.dataflow.dflow INFO: Task 807 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 25c18acce806e87dc2b824647ffbad44 with result from task 807
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:08 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow DEBUG: Task 807 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 808 has memoization hash e3067e41090f5a05a07d7b54ea10adbe
parsl.dataflow.memoization INFO: Task 808 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158956c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 808 try 0 launched on executor multi with executor id 692
parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/5eab55d6-136e-4bea-bcb9-400e62ac6758_characterizeImage_855910_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/5eab55d6-136e-4bea-bcb9-400e62ac6758_characterizeImage_855910_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/e8911895-0ef8-4fb8-9282-06daed1ffff6_isr_855910_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/e8911895-0ef8-4fb8-9282-06daed1ffff6_isr_855910_44.stderr
parsl.dataflow.dflow INFO: Task 795 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2e004977c5663572995a3a1fde9ea3c2 with result from task 795
parsl.dataflow.dflow DEBUG: Task 795 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 796 has memoization hash 0595e7baa35dcec787b5608967f1b535
parsl.dataflow.memoization INFO: Task 796 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15894a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 796 try 0 launched on executor multi with executor id 693
parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c3160f01-f18d-4235-be9b-9e9d350031b0_characterizeImage_855820_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c3160f01-f18d-4235-be9b-9e9d350031b0_characterizeImage_855820_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/38d16b36-3eb8-4cb1-93a5-b8b065e85842_isr_855820_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/38d16b36-3eb8-4cb1-93a5-b8b065e85842_isr_855820_50.stderr
parsl.dataflow.dflow INFO: Task 1008 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d887ff5ae7fe11406491a3d1389d5aea with result from task 1008
parsl.dataflow.dflow DEBUG: Task 1008 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1009 has memoization hash b0715db9d5e46d9fa36de467df3d1638
parsl.dataflow.memoization INFO: Task 1009 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1009 try 0 launched on executor multi with executor id 694
parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a0faff6d-ecca-4486-82d5-3c0f1176aa05_characterizeImage_855822_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a0faff6d-ecca-4486-82d5-3c0f1176aa05_characterizeImage_855822_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/74e71e55-77a6-4c79-8db5-e759865a3b5a_isr_855822_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/74e71e55-77a6-4c79-8db5-e759865a3b5a_isr_855822_29.stderr
parsl.dataflow.dflow INFO: Task 1023 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9c88f9c5b937eb44dcb3a1f041163e07 with result from task 1023
parsl.dataflow.dflow DEBUG: Task 1023 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1024 has memoization hash b33a5f85182b243afd76e48b673a881b
parsl.dataflow.memoization INFO: Task 1024 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1024 try 0 launched on executor multi with executor id 695
parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/f0d8d226-e2c1-4d10-9c29-2297e345f592_characterizeImage_855909_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/f0d8d226-e2c1-4d10-9c29-2297e345f592_characterizeImage_855909_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/6499acfb-9467-4273-b59f-479f31254b9c_isr_855909_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/6499acfb-9467-4273-b59f-479f31254b9c_isr_855909_19.stderr
parsl.dataflow.dflow INFO: Task 1011 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8f531f9422fa2f1ecdb2912c952fd7cf with result from task 1011
parsl.dataflow.dflow DEBUG: Task 1011 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1012 has memoization hash f39c1b0279745604638f70de66672693
parsl.dataflow.memoization INFO: Task 1012 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1012 try 0 launched on executor multi with executor id 696
parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b61b8bb7-c663-42e6-becd-9b2ec397287a_characterizeImage_855910_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b61b8bb7-c663-42e6-becd-9b2ec397287a_characterizeImage_855910_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/d923c619-47b8-4965-9e17-31d9609283e2_isr_855910_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/d923c619-47b8-4965-9e17-31d9609283e2_isr_855910_42.stderr
parsl.dataflow.dflow INFO: Task 1017 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e168c1bf7e81f85fb9820487c8dbb6b3 with result from task 1017
parsl.dataflow.dflow DEBUG: Task 1017 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1018 has memoization hash f4ee73b47b8cff4e6a4c1a1987a98295
parsl.dataflow.memoization INFO: Task 1018 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1018 try 0 launched on executor multi with executor id 697
parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a57c1a8e-c630-4041-a7b0-476fc439662a_characterizeImage_855822_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a57c1a8e-c630-4041-a7b0-476fc439662a_characterizeImage_855822_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/8dff2a86-baa3-4bc9-9a27-8de6a73ae0db_isr_855822_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/8dff2a86-baa3-4bc9-9a27-8de6a73ae0db_isr_855822_5.stderr
parsl.dataflow.dflow INFO: Task 1014 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 65a0a70986d838fc1ba0484682d6d8a0 with result from task 1014
parsl.dataflow.dflow DEBUG: Task 1014 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1015 has memoization hash 4688c2bb8421a430df0b203449375278
parsl.dataflow.memoization INFO: Task 1015 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1015 try 0 launched on executor multi with executor id 698
parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/658ecdf0-5d3c-411b-b98e-69b32732774e_characterizeImage_855909_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/658ecdf0-5d3c-411b-b98e-69b32732774e_characterizeImage_855909_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/38bac66a-25d7-4d7f-a59f-ccc61dfd833d_isr_855909_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/38bac66a-25d7-4d7f-a59f-ccc61dfd833d_isr_855909_6.stderr
parsl.dataflow.dflow INFO: Task 897 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c1bcbd075144e6786070da979c68ef45 with result from task 897
parsl.dataflow.dflow DEBUG: Task 897 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 898 has memoization hash d2254b6fbc4f03b3296d8ef5af7b17a2
parsl.dataflow.memoization INFO: Task 898 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156eb4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 898 try 0 launched on executor multi with executor id 699
parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/ea1ab206-74a7-46ee-bd6b-e656cbde3137_characterizeImage_855910_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/ea1ab206-74a7-46ee-bd6b-e656cbde3137_characterizeImage_855910_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/6c4fb8de-1280-46e5-834f-f7e270fc8754_isr_855910_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/6c4fb8de-1280-46e5-834f-f7e270fc8754_isr_855910_43.stderr
parsl.dataflow.dflow INFO: Task 1026 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6d4390f66cb2f74ed79ac5e8dfb3abcd with result from task 1026
parsl.dataflow.dflow DEBUG: Task 1026 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1027 has memoization hash b8a4bb9a705eb094849c37e9bd360631
parsl.dataflow.memoization INFO: Task 1027 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1027 try 0 launched on executor multi with executor id 700
parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/f2cbd907-26f7-4870-aa5b-2c38f3c42fa5_characterizeImage_855909_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/f2cbd907-26f7-4870-aa5b-2c38f3c42fa5_characterizeImage_855909_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/8a18c895-bd26-4245-9022-4cbb8518ad05_isr_855909_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/8a18c895-bd26-4245-9022-4cbb8518ad05_isr_855909_15.stderr
parsl.dataflow.dflow INFO: Task 1005 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f18785e410e0a1a1d69fc895803c862b with result from task 1005
parsl.dataflow.dflow DEBUG: Task 1005 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1006 has memoization hash 411574b3c89a389b37bffc547f24cf09
parsl.dataflow.memoization INFO: Task 1006 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1006 try 0 launched on executor multi with executor id 701
parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/aec76112-4e1f-4dbc-8089-7e8d0f06bb8a_characterizeImage_855909_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/aec76112-4e1f-4dbc-8089-7e8d0f06bb8a_characterizeImage_855909_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/ae8f990f-7450-4571-999e-34b1c52c2b89_isr_855909_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/ae8f990f-7450-4571-999e-34b1c52c2b89_isr_855909_21.stderr
parsl.dataflow.dflow INFO: Task 1020 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 03a3d33050f870348130b4c289dd21ff with result from task 1020
parsl.dataflow.dflow DEBUG: Task 1020 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1021 has memoization hash ebe691a9e4fa0961323d3610770dfeb5
parsl.dataflow.memoization INFO: Task 1021 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1021 try 0 launched on executor multi with executor id 702
parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/ae907f24-237b-43f1-93d0-2d392015e674_characterizeImage_855910_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/ae907f24-237b-43f1-93d0-2d392015e674_characterizeImage_855910_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/387f12b4-4060-4daa-a361-798b573c91e9_isr_855910_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/387f12b4-4060-4daa-a361-798b573c91e9_isr_855910_10.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 382 try 0 failed
parsl.dataflow.dflow INFO: Task 382 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9385d400-89ab-4c09-9287-eb72b6bb36a3_characterizeImage_855910_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9385d400-89ab-4c09-9287-eb72b6bb36a3_characterizeImage_855910_33.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 382 has memoization hash cf3065ae5f0e177351fb8bb10e157752
parsl.dataflow.memoization INFO: Task 382 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 382 try 1 launched on executor multi with executor id 703
parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9385d400-89ab-4c09-9287-eb72b6bb36a3_characterizeImage_855910_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9385d400-89ab-4c09-9287-eb72b6bb36a3_characterizeImage_855910_33.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 511 try 0 failed
parsl.dataflow.dflow INFO: Task 511 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/0fa21b17-4bee-4749-be6f-82640d5b659f_characterizeImage_855911_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/0fa21b17-4bee-4749-be6f-82640d5b659f_characterizeImage_855911_28.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 511 has memoization hash c8f242c7b9e33d0220c71832797d4823
parsl.dataflow.memoization INFO: Task 511 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15946020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 511 try 1 launched on executor multi with executor id 704
parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/0fa21b17-4bee-4749-be6f-82640d5b659f_characterizeImage_855911_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/0fa21b17-4bee-4749-be6f-82640d5b659f_characterizeImage_855911_28.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 370 try 0 failed
parsl.dataflow.dflow INFO: Task 370 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/17d4071f-0f92-4732-8a33-e6d45e1b118d_characterizeImage_855911_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/17d4071f-0f92-4732-8a33-e6d45e1b118d_characterizeImage_855911_19.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 370 has memoization hash c01a7dbf4f56fa06026a82b866dccf1d
parsl.dataflow.memoization INFO: Task 370 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 370 try 1 launched on executor multi with executor id 705
parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/17d4071f-0f92-4732-8a33-e6d45e1b118d_characterizeImage_855911_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/17d4071f-0f92-4732-8a33-e6d45e1b118d_characterizeImage_855911_19.stderr
parsl.dataflow.dflow INFO: Task 816 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry da2318befdf94b972210c76213407f7a with result from task 816
parsl.dataflow.dflow DEBUG: Task 816 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 817 has memoization hash d030e67f2a06573c35fce35c559b1928
parsl.dataflow.memoization INFO: Task 817 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15896020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 817 try 0 launched on executor multi with executor id 706
parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/f2c912ed-3c21-466e-8fe8-2479c91e6172_characterizeImage_855820_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/f2c912ed-3c21-466e-8fe8-2479c91e6172_characterizeImage_855820_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0d3db246-cee8-4a78-8b89-6bf1dd319a3c_isr_855820_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/0d3db246-cee8-4a78-8b89-6bf1dd319a3c_isr_855820_40.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 373 try 0 failed
parsl.dataflow.dflow INFO: Task 373 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/5f20bc73-308d-49d7-b631-f3754c89a722_characterizeImage_855910_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/5f20bc73-308d-49d7-b631-f3754c89a722_characterizeImage_855910_32.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 373 has memoization hash cb69a8e81091c970c23065e7690154af
parsl.dataflow.memoization INFO: Task 373 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 373 try 1 launched on executor multi with executor id 707
parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/5f20bc73-308d-49d7-b631-f3754c89a722_characterizeImage_855910_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/5f20bc73-308d-49d7-b631-f3754c89a722_characterizeImage_855910_32.stderr
parsl.dataflow.dflow INFO: Task 876 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cfe5ee8af5cf0e6e003efba3e6f0281b with result from task 876
parsl.dataflow.dflow DEBUG: Task 876 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 877 has memoization hash d1450f298434df98b4971908452b053e
parsl.dataflow.memoization INFO: Task 877 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 877 try 0 launched on executor multi with executor id 708
parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/9fad771b-2ef1-4846-b514-777fe8f70f6a_characterizeImage_855909_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/9fad771b-2ef1-4846-b514-777fe8f70f6a_characterizeImage_855909_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/74499a47-312a-4ba3-9e2b-0c22d74b192d_isr_855909_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/74499a47-312a-4ba3-9e2b-0c22d74b192d_isr_855909_30.stderr
parsl.dataflow.dflow INFO: Task 873 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 090185391210d609d94164a08c269b45 with result from task 873
parsl.dataflow.dflow DEBUG: Task 873 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 874 has memoization hash 2b83b6fb26d8ad5d25fd10096cc035eb
parsl.dataflow.memoization INFO: Task 874 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 874 try 0 launched on executor multi with executor id 709
parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/fdd3420a-55ad-4acd-b343-b131844ced09_characterizeImage_855822_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/fdd3420a-55ad-4acd-b343-b131844ced09_characterizeImage_855822_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/8623eb92-f4fd-4c1f-a6cd-b27ee2b42d46_isr_855822_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/8623eb92-f4fd-4c1f-a6cd-b27ee2b42d46_isr_855822_21.stderr
parsl.dataflow.dflow INFO: Task 870 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 600b00667825d193caab48278a3132eb with result from task 870
parsl.dataflow.dflow DEBUG: Task 870 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 871 has memoization hash fef774e2c038f71ef37c60fc6a3b7190
parsl.dataflow.memoization INFO: Task 871 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 871 try 0 launched on executor multi with executor id 710
parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a3fa3bf8-5523-4fc9-adb3-19d427cb862a_characterizeImage_855822_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a3fa3bf8-5523-4fc9-adb3-19d427cb862a_characterizeImage_855822_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/9813a636-51b6-40c5-b797-d49eda87b010_isr_855822_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/9813a636-51b6-40c5-b797-d49eda87b010_isr_855822_32.stderr
parsl.dataflow.dflow INFO: Task 885 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3c6eea54d4616596b5ec2ba6c0616f8e with result from task 885
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:08 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow DEBUG: Task 885 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 886 has memoization hash 3cf7610271a5d288ac714317d3900ddc
parsl.dataflow.memoization INFO: Task 886 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156ea840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 886 try 0 launched on executor multi with executor id 711
parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/212764aa-eb57-400c-b873-c889eec56055_characterizeImage_855909_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/212764aa-eb57-400c-b873-c889eec56055_characterizeImage_855909_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/aaba556b-2031-41af-855e-aba7fb81e0e1_isr_855909_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/aaba556b-2031-41af-855e-aba7fb81e0e1_isr_855909_1.stderr
parsl.dataflow.dflow INFO: Task 891 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c2934d1dc98920086c05d831d2b95245 with result from task 891
parsl.dataflow.dflow DEBUG: Task 891 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 892 has memoization hash 8ae029a88de79497dba0fa54e6668f39
parsl.dataflow.memoization INFO: Task 892 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156eae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 892 try 0 launched on executor multi with executor id 712
parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ddf3813f-0eac-4f77-be55-98a088d2ddf0_characterizeImage_855821_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ddf3813f-0eac-4f77-be55-98a088d2ddf0_characterizeImage_855821_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/d45c8313-e5cb-48c7-9e8c-197d2464c024_isr_855821_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/d45c8313-e5cb-48c7-9e8c-197d2464c024_isr_855821_40.stderr
parsl.dataflow.dflow INFO: Task 804 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9cdf70246072a3645efd02b6161307d3 with result from task 804
parsl.dataflow.dflow DEBUG: Task 804 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 805 has memoization hash b67a59e1b455bab70309210d572dbcfc
parsl.dataflow.memoization INFO: Task 805 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158953a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 805 try 0 launched on executor multi with executor id 713
parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/178fe260-4ff9-496a-86e6-716a79e9429a_characterizeImage_855821_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/178fe260-4ff9-496a-86e6-716a79e9429a_characterizeImage_855821_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/ace894e0-3704-445b-97c5-b2173f2fcae1_isr_855821_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/ace894e0-3704-445b-97c5-b2173f2fcae1_isr_855821_26.stderr
parsl.dataflow.dflow INFO: Task 813 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 00dd9ff12ff07d73585f5701b6357058 with result from task 813
parsl.dataflow.dflow DEBUG: Task 813 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 814 has memoization hash 588483d56ff79181df4bde4a61324f48
parsl.dataflow.memoization INFO: Task 814 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15895d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 814 try 0 launched on executor multi with executor id 714
parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/4b4873fc-b96f-490b-a882-0d8ddedf4296_characterizeImage_855821_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/4b4873fc-b96f-490b-a882-0d8ddedf4296_characterizeImage_855821_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/bdfa5a9c-b1e0-41d2-b3f7-6c4f5912873c_isr_855821_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/bdfa5a9c-b1e0-41d2-b3f7-6c4f5912873c_isr_855821_25.stderr
parsl.dataflow.dflow INFO: Task 903 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b2552f9d1fcacd747727e886e334f1c9 with result from task 903
parsl.dataflow.dflow DEBUG: Task 903 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 904 has memoization hash da190e75fac59397d9d809b73dd5476e
parsl.dataflow.memoization INFO: Task 904 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156ebb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 904 try 0 launched on executor multi with executor id 715
parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/313bd6af-fee3-4ede-ab6b-527851c5b361_characterizeImage_855822_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/313bd6af-fee3-4ede-ab6b-527851c5b361_characterizeImage_855822_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/fd2c77ba-9fc3-44cf-ae64-d70c1c767fe9_isr_855822_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/fd2c77ba-9fc3-44cf-ae64-d70c1c767fe9_isr_855822_61.stderr
parsl.dataflow.dflow INFO: Task 879 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5e055ef1c20db1b5878d899ac4f0ea55 with result from task 879
parsl.dataflow.dflow DEBUG: Task 879 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 880 has memoization hash 7e3c42ad9a9e03f0d0678ba7c262aba9
parsl.dataflow.memoization INFO: Task 880 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156ea200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 880 try 0 launched on executor multi with executor id 716
parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/e5ba6904-f091-4fac-80d6-6804e9224d30_characterizeImage_855911_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/e5ba6904-f091-4fac-80d6-6804e9224d30_characterizeImage_855911_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a29dcab2-64fb-4816-91fb-016056abc58b_isr_855911_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a29dcab2-64fb-4816-91fb-016056abc58b_isr_855911_36.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 310 try 0 failed
parsl.dataflow.dflow INFO: Task 310 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/19e593de-557a-4ee4-b389-d28b2d53cc5e_characterizeImage_855911_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/19e593de-557a-4ee4-b389-d28b2d53cc5e_characterizeImage_855911_21.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 116741d08cc736010feb115f1d32f12f
parsl.dataflow.memoization INFO: Task 310 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 310 try 1 launched on executor multi with executor id 717
parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/19e593de-557a-4ee4-b389-d28b2d53cc5e_characterizeImage_855911_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/19e593de-557a-4ee4-b389-d28b2d53cc5e_characterizeImage_855911_21.stderr
parsl.dataflow.dflow INFO: Task 810 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 34c34a15909074867e581dedc1d26c23 with result from task 810
parsl.dataflow.dflow DEBUG: Task 810 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 811 has memoization hash cf75a9fc448acef23c62ece03e93effe
parsl.dataflow.memoization INFO: Task 811 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158959e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 811 try 0 launched on executor multi with executor id 718
parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/7ce92523-1adf-49ac-bebc-1a6d3e1431d7_characterizeImage_855822_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/7ce92523-1adf-49ac-bebc-1a6d3e1431d7_characterizeImage_855822_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/ba3b59c5-4ca5-4afd-930c-89be90c56dda_isr_855822_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/ba3b59c5-4ca5-4afd-930c-89be90c56dda_isr_855822_12.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 649 try 0 failed
parsl.dataflow.dflow INFO: Task 649 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/b86f8dcf-7ba5-4429-b66e-65fba80f6930_characterizeImage_855911_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/b86f8dcf-7ba5-4429-b66e-65fba80f6930_characterizeImage_855911_12.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 649 has memoization hash 6c16083a6bc22a33f459496b5e9d982d
parsl.dataflow.memoization INFO: Task 649 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 649 try 1 launched on executor multi with executor id 719
parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/b86f8dcf-7ba5-4429-b66e-65fba80f6930_characterizeImage_855911_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/b86f8dcf-7ba5-4429-b66e-65fba80f6930_characterizeImage_855911_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:09 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 1071 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 894996dd2dee8346d1113021a37e4ff4 with result from task 1071
parsl.dataflow.dflow DEBUG: Task 1071 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1072 has memoization hash 4d110a22fd0f938163de43e6f2161390
parsl.dataflow.memoization INFO: Task 1072 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1072 try 0 launched on executor multi with executor id 720
parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a0902891-8e8e-4426-96ef-365db6e697c5_characterizeImage_855910_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a0902891-8e8e-4426-96ef-365db6e697c5_characterizeImage_855910_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/2ede4735-0950-497c-a70b-5316d01523bd_isr_855910_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/2ede4735-0950-497c-a70b-5316d01523bd_isr_855910_26.stderr
parsl.dataflow.dflow INFO: Task 1062 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 031d0f751d155af1b41e4a26e21eb252 with result from task 1062
parsl.dataflow.dflow DEBUG: Task 1062 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1063 has memoization hash 7b8b2b324509518edf6d86e092304f04
parsl.dataflow.memoization INFO: Task 1063 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1063 try 0 launched on executor multi with executor id 721
parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/b597245e-4743-4e60-bea7-c3fd021ba5dd_characterizeImage_855821_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/b597245e-4743-4e60-bea7-c3fd021ba5dd_characterizeImage_855821_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/6995e03f-f5cc-49e0-815e-ae3d7e093079_isr_855821_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/6995e03f-f5cc-49e0-815e-ae3d7e093079_isr_855821_17.stderr
parsl.dataflow.dflow INFO: Task 1065 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ba30bc03a8bff5e4a957a8217ad4f41d with result from task 1065
parsl.dataflow.dflow DEBUG: Task 1065 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1066 has memoization hash 2b63b3ea86b8cacadcbef96c383d2d9a
parsl.dataflow.memoization INFO: Task 1066 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1066 try 0 launched on executor multi with executor id 722
parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/2b37cb8b-6722-4cd7-941a-c0f5f7533a61_characterizeImage_855910_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/2b37cb8b-6722-4cd7-941a-c0f5f7533a61_characterizeImage_855910_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/cf761848-8b9a-4732-a8d9-9419a6e935cf_isr_855910_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/cf761848-8b9a-4732-a8d9-9419a6e935cf_isr_855910_31.stderr
parsl.dataflow.dflow INFO: Task 1077 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 466dedf4d6215d543575cef014fc39ca with result from task 1077
parsl.dataflow.dflow DEBUG: Task 1077 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1078 has memoization hash e16c575941579d1b76a8d83bedfce084
parsl.dataflow.memoization INFO: Task 1078 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1078 try 0 launched on executor multi with executor id 723
parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c859df67-1e74-4492-980c-cf360868b857_characterizeImage_855911_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c859df67-1e74-4492-980c-cf360868b857_characterizeImage_855911_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/252286b1-b28f-4857-be29-06bc8a0f74fa_isr_855911_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/252286b1-b28f-4857-be29-06bc8a0f74fa_isr_855911_9.stderr
parsl.dataflow.dflow INFO: Task 1083 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 265802a4864d78d954b5928a3ec9b704 with result from task 1083
parsl.dataflow.dflow DEBUG: Task 1083 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1084 has memoization hash 88b6652c35afe8a5ae134781771ccd31
parsl.dataflow.memoization INFO: Task 1084 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1084 try 0 launched on executor multi with executor id 724
parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/fc5056c5-506c-4a84-9013-750786a510ef_characterizeImage_855822_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/fc5056c5-506c-4a84-9013-750786a510ef_characterizeImage_855822_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/2386f827-d9d7-43c3-9201-36565f5391a2_isr_855822_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/2386f827-d9d7-43c3-9201-36565f5391a2_isr_855822_28.stderr
parsl.dataflow.dflow INFO: Task 1095 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f58c17aa9ddd7e9b57eeea70c88d6e83 with result from task 1095
parsl.dataflow.dflow DEBUG: Task 1095 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1096 has memoization hash b7ba27bc69650476e62eb444e9b7a9f9
parsl.dataflow.memoization INFO: Task 1096 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1560c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1096 try 0 launched on executor multi with executor id 725
parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c6f9712a-9e9c-4a22-981f-a183a4c8dabe_characterizeImage_855820_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c6f9712a-9e9c-4a22-981f-a183a4c8dabe_characterizeImage_855820_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/23e7ec59-1506-49eb-82bf-a63464c2a073_isr_855820_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/23e7ec59-1506-49eb-82bf-a63464c2a073_isr_855820_22.stderr
parsl.dataflow.dflow INFO: Task 1086 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 28fa4c73fec77fcf8669195cac2d3f5b with result from task 1086
parsl.dataflow.dflow DEBUG: Task 1086 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1087 has memoization hash 99763c6e495d1ce754bcc3ef8d3ee80b
parsl.dataflow.memoization INFO: Task 1087 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1087 try 0 launched on executor multi with executor id 726
parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/01412f1c-5ed0-49ad-87c6-74536c0cc35a_characterizeImage_855821_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/01412f1c-5ed0-49ad-87c6-74536c0cc35a_characterizeImage_855821_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/3ecd8a7d-a861-4335-9c17-6a784c363614_isr_855821_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/3ecd8a7d-a861-4335-9c17-6a784c363614_isr_855821_62.stderr
parsl.dataflow.dflow INFO: Task 1089 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2019765511fbd0a99dd475d4c473969f with result from task 1089
parsl.dataflow.dflow DEBUG: Task 1089 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1090 has memoization hash 49d52aa7747a92804d822c4d51818118
parsl.dataflow.memoization INFO: Task 1090 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1090 try 0 launched on executor multi with executor id 727
parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/95761f5c-07cf-4119-ba16-224e290484e3_characterizeImage_855820_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/95761f5c-07cf-4119-ba16-224e290484e3_characterizeImage_855820_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/768a8a59-1444-4964-b81f-712074234c8e_isr_855820_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/768a8a59-1444-4964-b81f-712074234c8e_isr_855820_14.stderr
parsl.dataflow.dflow INFO: Task 906 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f5fe4785191b5decb55f4ab7e3e14465 with result from task 906
parsl.dataflow.dflow DEBUG: Task 906 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 907 has memoization hash 99bfb7181ce36dfeee9e52377cd110ac
parsl.dataflow.memoization INFO: Task 907 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156ebe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 907 try 0 launched on executor multi with executor id 728
parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7c41e0e5-1a0d-4bd6-b22c-9f6e89d0adfe_characterizeImage_855910_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7c41e0e5-1a0d-4bd6-b22c-9f6e89d0adfe_characterizeImage_855910_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/9258bda4-91e7-4758-bd5b-596e6e9debbd_isr_855910_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/9258bda4-91e7-4758-bd5b-596e6e9debbd_isr_855910_54.stderr
parsl.dataflow.dflow INFO: Task 1080 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a7394b50ccb9928e9ee85986a626ae28 with result from task 1080
parsl.dataflow.dflow DEBUG: Task 1080 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1081 has memoization hash b8b4a7b0708faafc466c57c8f744bf91
parsl.dataflow.memoization INFO: Task 1081 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1081 try 0 launched on executor multi with executor id 729
parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/5f330de0-4063-4425-a305-c24b04ae3132_characterizeImage_855820_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/5f330de0-4063-4425-a305-c24b04ae3132_characterizeImage_855820_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/9d79b5c0-1302-4222-84aa-294315783e8c_isr_855820_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/9d79b5c0-1302-4222-84aa-294315783e8c_isr_855820_49.stderr
parsl.dataflow.dflow INFO: Task 1098 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0672afa3a2fb647706aee055770f87c4 with result from task 1098
parsl.dataflow.dflow DEBUG: Task 1098 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1099 has memoization hash e4db02eaba30ff561f96874924f3f0ae
parsl.dataflow.memoization INFO: Task 1099 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1560c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1099 try 0 launched on executor multi with executor id 730
parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a0b55d1a-c76e-448e-8caf-f7efcd1e582e_characterizeImage_855910_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a0b55d1a-c76e-448e-8caf-f7efcd1e582e_characterizeImage_855910_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/ede0b153-4c6b-457f-9518-d480953443a9_isr_855910_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/ede0b153-4c6b-457f-9518-d480953443a9_isr_855910_18.stderr
parsl.dataflow.dflow INFO: Task 1092 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6751151623c655e1ae7e751db1d8f2d5 with result from task 1092
parsl.dataflow.dflow DEBUG: Task 1092 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1093 has memoization hash 984e331893d7d3f515bb828385c67c7b
parsl.dataflow.memoization INFO: Task 1093 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1093 try 0 launched on executor multi with executor id 731
parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/f683499d-05c9-46f7-8b35-32c3c36685d5_characterizeImage_855820_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/f683499d-05c9-46f7-8b35-32c3c36685d5_characterizeImage_855820_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/01404079-48a7-4757-b2d7-dc5816702d15_isr_855820_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/01404079-48a7-4757-b2d7-dc5816702d15_isr_855820_4.stderr
parsl.dataflow.dflow INFO: Task 798 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2cc151ebe8d7d14ef1369a5471097e92 with result from task 798
parsl.dataflow.dflow DEBUG: Task 798 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 799 has memoization hash dffcc47853e66040cbb0f8c406308654
parsl.dataflow.memoization INFO: Task 799 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15894d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 799 try 0 launched on executor multi with executor id 732
parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/8083e6ed-55c2-4346-bb73-e777b6e2ee60_characterizeImage_855909_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/8083e6ed-55c2-4346-bb73-e777b6e2ee60_characterizeImage_855909_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/870fedf0-b3ff-48bb-a501-da12b5771fe1_isr_855909_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/870fedf0-b3ff-48bb-a501-da12b5771fe1_isr_855909_3.stderr
parsl.dataflow.dflow INFO: Task 993 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9a7861470c4f7969398bf6753dfacf13 with result from task 993
parsl.dataflow.dflow DEBUG: Task 993 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 994 has memoization hash 34ec087cd2d861097b19c3c728dbeb49
parsl.dataflow.memoization INFO: Task 994 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 994 try 0 launched on executor multi with executor id 733
parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/81cf190f-6b54-48ff-96e9-87e9e07a6928_characterizeImage_855910_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/81cf190f-6b54-48ff-96e9-87e9e07a6928_characterizeImage_855910_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/f64bebbb-f618-490b-a72b-365abe74fe95_isr_855910_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/f64bebbb-f618-490b-a72b-365abe74fe95_isr_855910_9.stderr
parsl.dataflow.dflow INFO: Task 894 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9529aa8dcf3cd3e8b5b54251d9cf4f46 with result from task 894
parsl.dataflow.dflow DEBUG: Task 894 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 895 has memoization hash 361af696bb66b227cee1094fb63d27e0
parsl.dataflow.memoization INFO: Task 895 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156eb1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 895 try 0 launched on executor multi with executor id 734
parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/b7760292-e48e-4788-95aa-882dd0522c01_characterizeImage_855820_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/b7760292-e48e-4788-95aa-882dd0522c01_characterizeImage_855820_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/5bd3a889-2528-41ba-8d35-e5c0411628ab_isr_855820_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/5bd3a889-2528-41ba-8d35-e5c0411628ab_isr_855820_60.stderr
parsl.dataflow.dflow INFO: Task 984 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 34c889da93b210397edef2771c57b605 with result from task 984
parsl.dataflow.dflow DEBUG: Task 984 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 985 has memoization hash 07e9e47bcfc831eff8f4d5e013c43d01
parsl.dataflow.memoization INFO: Task 985 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 985 try 0 launched on executor multi with executor id 735
parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c92cc6f5-cf4f-4d64-9779-b90e3f42d0fc_characterizeImage_855911_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c92cc6f5-cf4f-4d64-9779-b90e3f42d0fc_characterizeImage_855911_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/1dce9fe6-80ee-4e34-a8d6-5cd1d4cacd6a_isr_855911_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/1dce9fe6-80ee-4e34-a8d6-5cd1d4cacd6a_isr_855911_58.stderr
parsl.dataflow.dflow INFO: Task 987 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2a4cc6b0c3ea0c1dba520ddd3199bb7a with result from task 987
parsl.dataflow.dflow DEBUG: Task 987 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 988 has memoization hash 5057527f2327a4cca4c80f235d754193
parsl.dataflow.memoization INFO: Task 988 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 988 try 0 launched on executor multi with executor id 736
parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/06755dea-f6ce-4dd0-a7f1-2923082f7246_characterizeImage_855820_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/06755dea-f6ce-4dd0-a7f1-2923082f7246_characterizeImage_855820_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/ab28f426-ca4c-4c9e-9300-a7cac91eb9ff_isr_855820_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/ab28f426-ca4c-4c9e-9300-a7cac91eb9ff_isr_855820_21.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 100 try 0 failed
parsl.dataflow.dflow INFO: Task 100 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/d5e4af58-c2aa-4647-bf1f-61b47cefb824_characterizeImage_855910_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/d5e4af58-c2aa-4647-bf1f-61b47cefb824_characterizeImage_855910_8.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 100 has memoization hash ca030864dfb5ab35069f670cb4ce2c19
parsl.dataflow.memoization INFO: Task 100 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 100 try 1 launched on executor multi with executor id 737
parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/d5e4af58-c2aa-4647-bf1f-61b47cefb824_characterizeImage_855910_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/d5e4af58-c2aa-4647-bf1f-61b47cefb824_characterizeImage_855910_8.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 103 try 0 failed
parsl.dataflow.dflow INFO: Task 103 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/f321db45-b3e0-4ce4-a456-38b579e7c1ff_characterizeImage_855911_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/f321db45-b3e0-4ce4-a456-38b579e7c1ff_characterizeImage_855911_38.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 36686cc28c22241ddbd9aaa70188db58
parsl.dataflow.memoization INFO: Task 103 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 103 try 1 launched on executor multi with executor id 738
parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/f321db45-b3e0-4ce4-a456-38b579e7c1ff_characterizeImage_855911_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/f321db45-b3e0-4ce4-a456-38b579e7c1ff_characterizeImage_855911_38.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:09 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 930 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1863bb1d4148036062e8016de42b88e3 with result from task 930
parsl.dataflow.dflow DEBUG: Task 930 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 931 has memoization hash 6562a1fe08263739b9edf44ee3ea57c6
parsl.dataflow.memoization INFO: Task 931 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15729760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 931 try 0 launched on executor multi with executor id 739
parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/f280d8c3-0982-4f29-a8ef-cc735bc3c4cc_characterizeImage_855909_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/f280d8c3-0982-4f29-a8ef-cc735bc3c4cc_characterizeImage_855909_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/486b8cd6-1343-44c2-bc63-789071c5f890_isr_855909_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/486b8cd6-1343-44c2-bc63-789071c5f890_isr_855909_32.stderr
parsl.dataflow.dflow INFO: Task 927 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8c16590766823ff836277edd3ba4e90f with result from task 927
parsl.dataflow.dflow DEBUG: Task 927 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 928 has memoization hash 5bfa23b8cc6bdbfc81d179573bae1585
parsl.dataflow.memoization INFO: Task 928 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15729440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 928 try 0 launched on executor multi with executor id 740
parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/539fcb88-d75b-4b6e-8708-537ee6010854_characterizeImage_855911_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/539fcb88-d75b-4b6e-8708-537ee6010854_characterizeImage_855911_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/2a8e4445-7b0d-477d-a1c0-99200431e2bf_isr_855911_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/2a8e4445-7b0d-477d-a1c0-99200431e2bf_isr_855911_5.stderr
parsl.dataflow.dflow INFO: Task 840 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a551d02c2db46c4975d4bf21726593dd with result from task 840
parsl.dataflow.dflow DEBUG: Task 840 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 841 has memoization hash 5c4bba1302ef44cb51e4638acc8d3ffe
parsl.dataflow.memoization INFO: Task 841 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15897920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 841 try 0 launched on executor multi with executor id 741
parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/afe3182f-93d7-45a7-8779-80cee2563e76_characterizeImage_855821_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/afe3182f-93d7-45a7-8779-80cee2563e76_characterizeImage_855821_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/9b1e7b79-7e11-41f1-9e64-6cf59602de77_isr_855821_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/9b1e7b79-7e11-41f1-9e64-6cf59602de77_isr_855821_42.stderr
parsl.dataflow.dflow INFO: Task 834 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a205f83b1d4648bdba5fc6948aab8896 with result from task 834
parsl.dataflow.dflow DEBUG: Task 834 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 835 has memoization hash 8103ff89eb5dc532afae57343911b635
parsl.dataflow.memoization INFO: Task 835 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158972e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 835 try 0 launched on executor multi with executor id 742
parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/9a8f8d87-b354-4593-9dfa-27d9f780b288_characterizeImage_855909_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/9a8f8d87-b354-4593-9dfa-27d9f780b288_characterizeImage_855909_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/21724746-21af-49bf-b740-1ddea7894017_isr_855909_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/21724746-21af-49bf-b740-1ddea7894017_isr_855909_57.stderr
parsl.dataflow.dflow INFO: Task 822 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 023f23a0cdee12fb623e971a57839ec3 with result from task 822
parsl.dataflow.dflow DEBUG: Task 822 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 823 has memoization hash f14718381899a1619e1cf3f09a95ef33
parsl.dataflow.memoization INFO: Task 823 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15896660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 823 try 0 launched on executor multi with executor id 743
parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8af071b6-eb9b-4d32-82ed-ce59eb90ebc7_characterizeImage_855911_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8af071b6-eb9b-4d32-82ed-ce59eb90ebc7_characterizeImage_855911_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/72d15fde-1e6b-4f59-9931-eb1620cfeb22_isr_855911_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/72d15fde-1e6b-4f59-9931-eb1620cfeb22_isr_855911_20.stderr
parsl.dataflow.dflow INFO: Task 831 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 10d011b734a1086a838d19a2a08c3320 with result from task 831
parsl.dataflow.dflow DEBUG: Task 831 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 832 has memoization hash 2da1a99999ec608860a7d755b9373590
parsl.dataflow.memoization INFO: Task 832 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15896fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 832 try 0 launched on executor multi with executor id 744
parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/d7110b2d-e20c-49a3-b4f7-47908b9769f6_characterizeImage_855910_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/d7110b2d-e20c-49a3-b4f7-47908b9769f6_characterizeImage_855910_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/8960334f-5330-49ea-95d3-0c6a61b656a6_isr_855910_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/8960334f-5330-49ea-95d3-0c6a61b656a6_isr_855910_14.stderr
parsl.dataflow.dflow INFO: Task 924 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 955fa11b8aed6ef81155f26c91d32ec9 with result from task 924
parsl.dataflow.dflow DEBUG: Task 924 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 925 has memoization hash 6fb2de3aca395efe2ff1b4a17db636af
parsl.dataflow.memoization INFO: Task 925 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15729120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 925 try 0 launched on executor multi with executor id 745
parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/08967090-c681-4b0d-83ba-c80fa931a75c_characterizeImage_855821_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/08967090-c681-4b0d-83ba-c80fa931a75c_characterizeImage_855821_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/47124ece-5fd4-4a43-9728-d71ff8eb5486_isr_855821_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/47124ece-5fd4-4a43-9728-d71ff8eb5486_isr_855821_60.stderr
parsl.dataflow.dflow INFO: Task 942 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7cc58aea66a0f81881becbb73b2a94be with result from task 942
parsl.dataflow.dflow DEBUG: Task 942 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 943 has memoization hash 58ecb3603abbc2064411e53d0e773887
parsl.dataflow.memoization INFO: Task 943 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 943 try 0 launched on executor multi with executor id 746
parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c2cb5987-4744-4a73-a11e-01711df145b9_characterizeImage_855911_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c2cb5987-4744-4a73-a11e-01711df145b9_characterizeImage_855911_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/46766e17-037c-4a45-a6ff-7a4205663c99_isr_855911_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/46766e17-037c-4a45-a6ff-7a4205663c99_isr_855911_8.stderr
parsl.dataflow.dflow INFO: Task 939 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5ce01f1c8d409de1d8882d759cb37d6a with result from task 939
parsl.dataflow.dflow DEBUG: Task 939 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 940 has memoization hash 31dfc87fecdbb2ffc6efb50c17cd23b4
parsl.dataflow.memoization INFO: Task 940 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 940 try 0 launched on executor multi with executor id 747
parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/9aa1ef21-e968-49df-89ad-1ceccc2ff409_characterizeImage_855911_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/9aa1ef21-e968-49df-89ad-1ceccc2ff409_characterizeImage_855911_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/0323c06f-e0ca-49a5-aa73-f9b70dd99997_isr_855911_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/0323c06f-e0ca-49a5-aa73-f9b70dd99997_isr_855911_47.stderr
parsl.dataflow.dflow INFO: Task 837 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fb6b7905d952f6caf75303793a3e5b5e with result from task 837
parsl.dataflow.dflow DEBUG: Task 837 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 838 has memoization hash 4cf55512fe3a301645efca070f28c4b6
parsl.dataflow.memoization INFO: Task 838 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15897600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 838 try 0 launched on executor multi with executor id 748
parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/5b42ec12-14a4-41cd-937e-29587c8eb31b_characterizeImage_855909_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/5b42ec12-14a4-41cd-937e-29587c8eb31b_characterizeImage_855909_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/4f8aae4b-766b-4746-ae16-4971ff0113d8_isr_855909_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/4f8aae4b-766b-4746-ae16-4971ff0113d8_isr_855909_44.stderr
parsl.dataflow.dflow INFO: Task 828 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 129e71ae831c765629d2c35671ee35e7 with result from task 828
parsl.dataflow.dflow DEBUG: Task 828 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 829 has memoization hash f0b50c199d967c0f4078ca0e38af84ee
parsl.dataflow.memoization INFO: Task 829 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15896ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 829 try 0 launched on executor multi with executor id 749
parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/baee5428-1d8a-4963-8c53-51c5e8aaf499_characterizeImage_855821_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/baee5428-1d8a-4963-8c53-51c5e8aaf499_characterizeImage_855821_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/8233cefd-094d-4686-becb-c052446e79b7_isr_855821_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/8233cefd-094d-4686-becb-c052446e79b7_isr_855821_47.stderr
parsl.dataflow.dflow INFO: Task 825 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 41e6a95a932b62838656526216fe34d8 with result from task 825
parsl.dataflow.dflow DEBUG: Task 825 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 826 has memoization hash 11701675ab424a79fe16d8818f374d21
parsl.dataflow.memoization INFO: Task 826 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15896980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 826 try 0 launched on executor multi with executor id 750
parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/5e4fdc11-62ac-42b8-b4e4-51150a991cef_characterizeImage_855820_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/5e4fdc11-62ac-42b8-b4e4-51150a991cef_characterizeImage_855820_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/6b8cfc49-c688-474d-9c9b-db5b00b80b85_isr_855820_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/6b8cfc49-c688-474d-9c9b-db5b00b80b85_isr_855820_56.stderr
parsl.dataflow.dflow INFO: Task 843 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 03ef3eef05b4f025b520215242194265 with result from task 843
parsl.dataflow.dflow DEBUG: Task 843 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 844 has memoization hash 30199eda9364927796332c124d0d782e
parsl.dataflow.memoization INFO: Task 844 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15897c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 844 try 0 launched on executor multi with executor id 751
parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/1ffe8752-e66b-4e61-9531-ed3c1f03b719_characterizeImage_855909_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/1ffe8752-e66b-4e61-9531-ed3c1f03b719_characterizeImage_855909_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/6a43c1e1-c745-4ad0-90d5-3e920098cf91_isr_855909_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/6a43c1e1-c745-4ad0-90d5-3e920098cf91_isr_855909_23.stderr
parsl.dataflow.dflow INFO: Task 933 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 16626109032b9c83f576662beea05b34 with result from task 933
parsl.dataflow.dflow DEBUG: Task 933 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 934 has memoization hash 17cec5ab4d994ed2c22832298dc57f52
parsl.dataflow.memoization INFO: Task 934 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15729a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 934 try 0 launched on executor multi with executor id 752
parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/14b2f87d-08ad-4d7b-a078-a8301b1539e3_characterizeImage_855821_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/14b2f87d-08ad-4d7b-a078-a8301b1539e3_characterizeImage_855821_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/58c5d8f3-d992-4fa2-87e9-279d413f8193_isr_855821_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/58c5d8f3-d992-4fa2-87e9-279d413f8193_isr_855821_4.stderr
parsl.dataflow.dflow INFO: Task 936 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 41f1e349aaf37a92f139563f5a114a45 with result from task 936
parsl.dataflow.dflow DEBUG: Task 936 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 937 has memoization hash 86156130ad2a683eee2e9487875854d4
parsl.dataflow.memoization INFO: Task 937 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15729da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 937 try 0 launched on executor multi with executor id 753
parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/23dafb7e-ace3-40b5-bbcb-60cbd204de5f_characterizeImage_855822_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/23dafb7e-ace3-40b5-bbcb-60cbd204de5f_characterizeImage_855822_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/1653afb0-f1f6-4d4e-acf7-eabfa30d7448_isr_855822_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/1653afb0-f1f6-4d4e-acf7-eabfa30d7448_isr_855822_24.stderr
parsl.dataflow.dflow INFO: Task 921 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 862078a634d1fd876ddddd6f2c27e566 with result from task 921
parsl.dataflow.dflow DEBUG: Task 921 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 922 has memoization hash 82454972e05f907affba19f45f0afb30
parsl.dataflow.memoization INFO: Task 922 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15728e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 922 try 0 launched on executor multi with executor id 754
parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/3c01eca9-482c-46e2-9f65-de2000bf934f_characterizeImage_855910_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/3c01eca9-482c-46e2-9f65-de2000bf934f_characterizeImage_855910_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/36cfd077-2fdc-4556-ad6d-a75a70d623ef_isr_855910_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/36cfd077-2fdc-4556-ad6d-a75a70d623ef_isr_855910_50.stderr
parsl.dataflow.dflow INFO: Task 966 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a3480e74e074fc1db4808d47f23e8859 with result from task 966
parsl.dataflow.dflow DEBUG: Task 966 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 967 has memoization hash 9b7a38a91c95c12090e05767346096f3
parsl.dataflow.memoization INFO: Task 967 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 967 try 0 launched on executor multi with executor id 755
parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/050feb2b-e2d5-4836-a789-83b143629ebb_characterizeImage_855822_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/050feb2b-e2d5-4836-a789-83b143629ebb_characterizeImage_855822_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/8a2ccb3c-c55f-430d-a07e-9da2b3d5d67a_isr_855822_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/8a2ccb3c-c55f-430d-a07e-9da2b3d5d67a_isr_855822_33.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 73 try 0 failed
parsl.dataflow.dflow INFO: Task 73 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/54458e78-9027-47ce-862b-4d889c61ef6f_characterizeImage_855910_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/54458e78-9027-47ce-862b-4d889c61ef6f_characterizeImage_855910_45.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 250d620d1601e63d735774a717cd6274
parsl.dataflow.memoization INFO: Task 73 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 73 try 1 launched on executor multi with executor id 756
parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/54458e78-9027-47ce-862b-4d889c61ef6f_characterizeImage_855910_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/54458e78-9027-47ce-862b-4d889c61ef6f_characterizeImage_855910_45.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 61 try 0 failed
parsl.dataflow.dflow INFO: Task 61 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/fd018433-8f13-4185-8dd5-1c952533989d_characterizeImage_855910_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/fd018433-8f13-4185-8dd5-1c952533989d_characterizeImage_855910_20.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 671678e604c26484b591d5d13831871e
parsl.dataflow.memoization INFO: Task 61 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 61 try 1 launched on executor multi with executor id 757
parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/fd018433-8f13-4185-8dd5-1c952533989d_characterizeImage_855910_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/fd018433-8f13-4185-8dd5-1c952533989d_characterizeImage_855910_20.stderr
parsl.dataflow.dflow INFO: Task 954 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5b5649b4c0a5093dbd005c8cd05b2a71 with result from task 954
parsl.dataflow.dflow DEBUG: Task 954 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 955 has memoization hash 3d4868080a7cff73eb491cefd6caba08
parsl.dataflow.memoization INFO: Task 955 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 955 try 0 launched on executor multi with executor id 758
parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/91f97023-ae22-4fb2-8b91-d0d1c119c4db_characterizeImage_855909_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/91f97023-ae22-4fb2-8b91-d0d1c119c4db_characterizeImage_855909_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/feb42d1c-a7b6-43c0-b170-31da5d4e70ae_isr_855909_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/feb42d1c-a7b6-43c0-b170-31da5d4e70ae_isr_855909_10.stderr
parsl.dataflow.dflow INFO: Task 963 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 81dee5fd781028e5f34486ff2b02b979 with result from task 963
parsl.dataflow.dflow DEBUG: Task 963 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 964 has memoization hash f518ef0c50dee16be2ee8528da7a6598
parsl.dataflow.memoization INFO: Task 964 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 964 try 0 launched on executor multi with executor id 759
parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/d2f137b1-2ecd-4710-a401-ff67c98329ec_characterizeImage_855909_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/d2f137b1-2ecd-4710-a401-ff67c98329ec_characterizeImage_855909_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/8ba65ba4-e46a-4444-bfe2-bbbb53c9dcbd_isr_855909_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/8ba65ba4-e46a-4444-bfe2-bbbb53c9dcbd_isr_855909_40.stderr
parsl.dataflow.dflow INFO: Task 948 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 58f9e27ef1d04b9bfa39d0a7696a4094 with result from task 948
parsl.dataflow.dflow DEBUG: Task 948 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 949 has memoization hash f42aecc1cfb330182a20374ddf1b7df6
parsl.dataflow.memoization INFO: Task 949 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 949 try 0 launched on executor multi with executor id 760
parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/c28f3221-e2f2-456c-9ceb-7b405ff67665_characterizeImage_855821_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/c28f3221-e2f2-456c-9ceb-7b405ff67665_characterizeImage_855821_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/cab0bc88-87a2-4c85-a46b-ad08242cca8e_isr_855821_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/cab0bc88-87a2-4c85-a46b-ad08242cca8e_isr_855821_50.stderr
parsl.dataflow.dflow INFO: Task 882 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 193db1ae23178d74a1181659b89523b6 with result from task 882
parsl.dataflow.dflow DEBUG: Task 882 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 883 has memoization hash 59caf018d23c6979f80fdf674ffdfd96
parsl.dataflow.memoization INFO: Task 883 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156ea520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 883 try 0 launched on executor multi with executor id 761
parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/73a09d07-6345-4188-a6bb-5b773b1ac1ea_characterizeImage_855910_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/73a09d07-6345-4188-a6bb-5b773b1ac1ea_characterizeImage_855910_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/7b31c6f5-0d7f-4c1a-8151-5afcbaedc581_isr_855910_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/7b31c6f5-0d7f-4c1a-8151-5afcbaedc581_isr_855910_38.stderr
parsl.dataflow.dflow INFO: Task 990 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ecf2b345c827f086048f974067b0c4d5 with result from task 990
parsl.dataflow.dflow DEBUG: Task 990 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 991 has memoization hash 4b2946970080e9d2108da395e5075440
parsl.dataflow.memoization INFO: Task 991 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 991 try 0 launched on executor multi with executor id 762
parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a36ab353-27f1-45b0-a927-5f2adda996cf_characterizeImage_855822_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a36ab353-27f1-45b0-a927-5f2adda996cf_characterizeImage_855822_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/232a0318-6b06-4300-92fd-a9165546e719_isr_855822_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/232a0318-6b06-4300-92fd-a9165546e719_isr_855822_34.stderr
parsl.dataflow.dflow INFO: Task 888 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 33d9ee338d5717b6893b5399d5872230 with result from task 888
parsl.dataflow.dflow DEBUG: Task 888 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 889 has memoization hash 51517bde30808a0da7ea8842fb638568
parsl.dataflow.memoization INFO: Task 889 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156eab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 889 try 0 launched on executor multi with executor id 763
parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/9aacd4d8-3585-4524-be65-b137cdda2dbb_characterizeImage_855820_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/9aacd4d8-3585-4524-be65-b137cdda2dbb_characterizeImage_855820_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/eaa53c28-6c30-48f4-9a87-ca71089d56b2_isr_855820_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/eaa53c28-6c30-48f4-9a87-ca71089d56b2_isr_855820_3.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:09 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:10 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 1053 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e0c477f5b4c829b878ef86c7d43d71bd with result from task 1053
parsl.dataflow.dflow DEBUG: Task 1053 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1054 has memoization hash 8be1e0b43e97c14935cd527cae6cd5a9
parsl.dataflow.memoization INFO: Task 1054 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1054 try 0 launched on executor multi with executor id 764
parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/beef6808-7a49-4e89-9015-95b28b6d89c4_characterizeImage_855821_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/beef6808-7a49-4e89-9015-95b28b6d89c4_characterizeImage_855821_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/5b528d9b-6e29-46ee-b653-d19982f317f7_isr_855821_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/5b528d9b-6e29-46ee-b653-d19982f317f7_isr_855821_1.stderr
parsl.dataflow.dflow INFO: Task 1056 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c2ea701eeaff248535aad108c508d4f8 with result from task 1056
parsl.dataflow.dflow DEBUG: Task 1056 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1057 has memoization hash 66c2f1977015bd213c1bd8023c993de9
parsl.dataflow.memoization INFO: Task 1057 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1057 try 0 launched on executor multi with executor id 765
parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/d3301da5-a335-4071-a778-ad8a68dba0c8_characterizeImage_855910_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/d3301da5-a335-4071-a778-ad8a68dba0c8_characterizeImage_855910_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/eb668132-1857-4ac5-906f-5f924c1d511d_isr_855910_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/eb668132-1857-4ac5-906f-5f924c1d511d_isr_855910_48.stderr
parsl.dataflow.dflow INFO: Task 981 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry be55ec840693439c10968da22a878891 with result from task 981
parsl.dataflow.dflow DEBUG: Task 981 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 982 has memoization hash 0aba1bb3007ec4bd33502f5057eb9307
parsl.dataflow.memoization INFO: Task 982 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 982 try 0 launched on executor multi with executor id 766
parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/bac4de84-eed6-4e4b-931b-1c29f3f689c8_characterizeImage_855911_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/bac4de84-eed6-4e4b-931b-1c29f3f689c8_characterizeImage_855911_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a716fc96-5512-4781-baa0-77afd728b2df_isr_855911_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/a716fc96-5512-4781-baa0-77afd728b2df_isr_855911_30.stderr
parsl.dataflow.dflow INFO: Task 975 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 31639962eec0682c736c58c450cdd56c with result from task 975
parsl.dataflow.dflow DEBUG: Task 975 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 976 has memoization hash 0c5c63b925090c1e642dbb652b95654e
parsl.dataflow.memoization INFO: Task 976 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 976 try 0 launched on executor multi with executor id 767
parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/2aac9e33-385b-4265-89fb-03d274996af0_characterizeImage_855820_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/2aac9e33-385b-4265-89fb-03d274996af0_characterizeImage_855820_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/e3c10186-a3da-403f-889f-7f17475d64d7_isr_855820_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/e3c10186-a3da-403f-889f-7f17475d64d7_isr_855820_46.stderr
parsl.dataflow.dflow INFO: Task 996 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 80ba1f86678d4d8d2df15c42282d6ee1 with result from task 996
parsl.dataflow.dflow DEBUG: Task 996 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 997 has memoization hash 7e7bb74245950934264139f22d6e6170
parsl.dataflow.memoization INFO: Task 997 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 997 try 0 launched on executor multi with executor id 768
parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/86659ec4-e2f0-47af-8918-e5bfc2fc579f_characterizeImage_855822_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/86659ec4-e2f0-47af-8918-e5bfc2fc579f_characterizeImage_855822_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/d1089740-bd29-4381-a280-c87153092af5_isr_855822_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/d1089740-bd29-4381-a280-c87153092af5_isr_855822_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:10 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 978 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 476e3361f7373bea6265c1feb709ab4d with result from task 978
parsl.dataflow.dflow DEBUG: Task 978 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 979 has memoization hash cec063361741c867881f5701030218c1
parsl.dataflow.memoization INFO: Task 979 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 979 try 0 launched on executor multi with executor id 769
parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/33fa17d6-d782-4649-9e77-241c861bcbae_characterizeImage_855911_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/33fa17d6-d782-4649-9e77-241c861bcbae_characterizeImage_855911_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/af20380d-5e9f-413e-8cad-812800e6f4f5_isr_855911_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855911/af20380d-5e9f-413e-8cad-812800e6f4f5_isr_855911_27.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 28 try 0 failed
parsl.dataflow.dflow INFO: Task 28 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/26eb6126-3dec-4e32-89ba-2f15a60704bb_characterizeImage_855910_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/26eb6126-3dec-4e32-89ba-2f15a60704bb_characterizeImage_855910_1.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 28 has memoization hash d65319fdaf240d1fd266d4fdc7251656
parsl.dataflow.memoization INFO: Task 28 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 28 try 1 launched on executor multi with executor id 770
parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/26eb6126-3dec-4e32-89ba-2f15a60704bb_characterizeImage_855910_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/26eb6126-3dec-4e32-89ba-2f15a60704bb_characterizeImage_855910_1.stderr
parsl.dataflow.dflow INFO: Task 945 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a17ac253ed58097e0f02b3f9ef8fdf7c with result from task 945
parsl.dataflow.dflow DEBUG: Task 945 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 946 has memoization hash 92dc1932e2f1d936dac08ede74616cee
parsl.dataflow.memoization INFO: Task 946 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 946 try 0 launched on executor multi with executor id 771
parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/960a4f8f-bdcb-4ab7-b3f8-91d3f67a398e_characterizeImage_855822_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/960a4f8f-bdcb-4ab7-b3f8-91d3f67a398e_characterizeImage_855822_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/0566f0cf-f47d-4089-a93a-19ef90867295_isr_855822_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/0566f0cf-f47d-4089-a93a-19ef90867295_isr_855822_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:10 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 960 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d624d200f779d21cb3f19cced9c4f74b with result from task 960
parsl.dataflow.dflow DEBUG: Task 960 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 961 has memoization hash 8bffe6cf04e0a6c585493ac80d07b880
parsl.dataflow.memoization INFO: Task 961 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 961 try 0 launched on executor multi with executor id 772
parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/47162c81-0123-4f42-afa5-49c1a2f5856b_characterizeImage_855909_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/47162c81-0123-4f42-afa5-49c1a2f5856b_characterizeImage_855909_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/59f7c88d-61c5-405e-8a3b-0a6fcc9cebf3_isr_855909_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855909/59f7c88d-61c5-405e-8a3b-0a6fcc9cebf3_isr_855909_42.stderr
parsl.dataflow.dflow INFO: Task 957 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9566d2ec235d10975fb28391ae4af4c4 with result from task 957
parsl.dataflow.dflow DEBUG: Task 957 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 958 has memoization hash 7acf68fddc916a95e5ca0642438c3d8d
parsl.dataflow.memoization INFO: Task 958 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 958 try 0 launched on executor multi with executor id 773
parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/bdd04c08-1946-4778-a3b6-1296bfc4264e_characterizeImage_855910_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/bdd04c08-1946-4778-a3b6-1296bfc4264e_characterizeImage_855910_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/3bf26ae9-c5fc-4364-96a3-dc8539da8bc8_isr_855910_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/3bf26ae9-c5fc-4364-96a3-dc8539da8bc8_isr_855910_61.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:11 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 325 try 0 failed
parsl.dataflow.dflow INFO: Task 325 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c665b36c-f2f1-4331-99fc-e27f91d5f12e_characterizeImage_855911_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c665b36c-f2f1-4331-99fc-e27f91d5f12e_characterizeImage_855911_61.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 07f04ed70709f8032682cf9daa92e3db
parsl.dataflow.memoization INFO: Task 325 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 325 try 1 launched on executor multi with executor id 774
parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c665b36c-f2f1-4331-99fc-e27f91d5f12e_characterizeImage_855911_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c665b36c-f2f1-4331-99fc-e27f91d5f12e_characterizeImage_855911_61.stderr
parsl.dataflow.dflow INFO: Task 951 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1703c9379a36b220ca51b2e9b2f43aa4 with result from task 951
parsl.dataflow.dflow DEBUG: Task 951 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 952 has memoization hash 4d89d0878e104f35f6dbe92d1881e4a3
parsl.dataflow.memoization INFO: Task 952 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 952 try 0 launched on executor multi with executor id 775
parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/252e616b-69e7-4fae-8213-4661f211ae07_characterizeImage_855910_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/252e616b-69e7-4fae-8213-4661f211ae07_characterizeImage_855910_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/6bd1bff9-1fff-4aa8-a44c-7b29dcfac192_isr_855910_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855910/6bd1bff9-1fff-4aa8-a44c-7b29dcfac192_isr_855910_29.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:11 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 577 try 0 failed
parsl.dataflow.dflow INFO: Task 577 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b0af1003-e9b5-405e-9c1e-712df5a6a216_characterizeImage_855910_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b0af1003-e9b5-405e-9c1e-712df5a6a216_characterizeImage_855910_16.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 577 has memoization hash d72d77aeba683cb553260f47f0ca9d24
parsl.dataflow.memoization INFO: Task 577 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 577 try 1 launched on executor multi with executor id 776
parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b0af1003-e9b5-405e-9c1e-712df5a6a216_characterizeImage_855910_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b0af1003-e9b5-405e-9c1e-712df5a6a216_characterizeImage_855910_16.stderr
parsl.dataflow.dflow INFO: Task 1074 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8bf80a11f72601965c8a22c36b7809e1 with result from task 1074
parsl.dataflow.dflow DEBUG: Task 1074 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1075 has memoization hash 323fe88ad0c9f24da10de84133a61175
parsl.dataflow.memoization INFO: Task 1075 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1075 try 0 launched on executor multi with executor id 777
parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/8213c3b3-9a1f-4fbd-a4ea-0d36d46029ce_characterizeImage_855820_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/8213c3b3-9a1f-4fbd-a4ea-0d36d46029ce_characterizeImage_855820_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/13d0bbad-12ca-4696-a159-500990c0dafe_isr_855820_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855820/13d0bbad-12ca-4696-a159-500990c0dafe_isr_855820_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:12 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 1105 try 0 failed
parsl.dataflow.dflow INFO: Task 1105 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/34a4e514-e396-44a4-8579-280c35a0e37e_characterizeImage_855910_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/34a4e514-e396-44a4-8579-280c35a0e37e_characterizeImage_855910_58.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1105 has memoization hash f4e736cebf1978a7b05b4b85446d4a4c
parsl.dataflow.memoization INFO: Task 1105 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1560cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1105 try 1 launched on executor multi with executor id 778
parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/34a4e514-e396-44a4-8579-280c35a0e37e_characterizeImage_855910_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/34a4e514-e396-44a4-8579-280c35a0e37e_characterizeImage_855910_58.stderr
parsl.dataflow.dflow INFO: Task 1068 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5497fd364b646ad094501e22de137513 with result from task 1068
parsl.dataflow.dflow DEBUG: Task 1068 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1069 has memoization hash bef6b320f73b3af41886fac2281572ff
parsl.dataflow.memoization INFO: Task 1069 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1069 try 0 launched on executor multi with executor id 779
parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ef62fb4f-64d0-4174-bac3-4504860186c9_characterizeImage_855821_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ef62fb4f-64d0-4174-bac3-4504860186c9_characterizeImage_855821_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/ac673ddf-c87f-4bd0-84be-e0163097f0ec_isr_855821_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855821/ac673ddf-c87f-4bd0-84be-e0163097f0ec_isr_855821_52.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 463 try 0 failed
parsl.dataflow.dflow INFO: Task 463 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/644a413f-2a07-4ae8-b7c0-8cc140b0e0d8_characterizeImage_855911_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/644a413f-2a07-4ae8-b7c0-8cc140b0e0d8_characterizeImage_855911_18.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 463 has memoization hash 5ddfe85142383fc284fb4790b66df0d4
parsl.dataflow.memoization INFO: Task 463 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 463 try 1 launched on executor multi with executor id 780
parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/644a413f-2a07-4ae8-b7c0-8cc140b0e0d8_characterizeImage_855911_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/644a413f-2a07-4ae8-b7c0-8cc140b0e0d8_characterizeImage_855911_18.stderr
parsl.dataflow.dflow INFO: Task 1059 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 64f0d1156183836ff98e741689908c75 with result from task 1059
parsl.dataflow.dflow DEBUG: Task 1059 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1060 has memoization hash 7f894eae81e630b563b8c12ba27b025f
parsl.dataflow.memoization INFO: Task 1060 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1060 try 0 launched on executor multi with executor id 781
parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/8d62c655-4f6c-4376-bcf8-167e5d25b8f0_characterizeImage_855822_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/8d62c655-4f6c-4376-bcf8-167e5d25b8f0_characterizeImage_855822_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/a4796649-3a06-4cba-a7b9-6f2fe1810990_isr_855822_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/isr/855822/a4796649-3a06-4cba-a7b9-6f2fe1810990_isr_855822_38.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:12 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:13 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 202 try 1 failed
parsl.dataflow.dflow ERROR: Task 202 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry c0d66659766a9cef44d1239f370f6c1f with result from task 202
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 203 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 203 try 0 failed
parsl.dataflow.dflow INFO: Task 203 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/b1ad7b97-056c-4b4b-87bf-342a37af19d5_calibrate_855911_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/b1ad7b97-056c-4b4b-87bf-342a37af19d5_calibrate_855911_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c5a4fe4e-d413-4790-808a-ade5a22926eb_characterizeImage_855911_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c5a4fe4e-d413-4790-808a-ade5a22926eb_characterizeImage_855911_2.stderr
parsl.dataflow.dflow INFO: Task 820 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8a07c89658bd0f5a7684eb22c162053c with result from task 820
parsl.dataflow.dflow DEBUG: Task 820 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 821 has memoization hash c2853a0c0390f71053fa2803f3a55208
parsl.dataflow.memoization INFO: Task 821 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15896160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 821 try 0 launched on executor multi with executor id 782
parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/874038c0-6129-4fb1-a3a4-1dad17c3ca65_calibrate_855911_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/874038c0-6129-4fb1-a3a4-1dad17c3ca65_calibrate_855911_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/b0348936-70a0-4820-b045-35897808d247_characterizeImage_855911_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/b0348936-70a0-4820-b045-35897808d247_characterizeImage_855911_50.stderr
parsl.dataflow.dflow INFO: Task 1000 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4a708be4d773e5e50e8c2eb2483ce45c with result from task 1000
parsl.dataflow.dflow DEBUG: Task 1000 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1001 has memoization hash 7cabc7b4f1da8c753a2e3846d3a95ba2
parsl.dataflow.memoization INFO: Task 1001 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1001 try 0 launched on executor multi with executor id 783
parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/851e6f2e-f498-4375-ba09-41b714b0c36b_calibrate_855911_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/851e6f2e-f498-4375-ba09-41b714b0c36b_calibrate_855911_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/3c4fcba1-3793-492e-8796-74c484951c00_characterizeImage_855911_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/3c4fcba1-3793-492e-8796-74c484951c00_characterizeImage_855911_35.stderr
parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3c0b44ad919eed72c767095649f49e1a with result from task 292
parsl.dataflow.dflow DEBUG: Task 292 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 293 has memoization hash c1340757feaff4f75202df85c1400c08
parsl.dataflow.memoization INFO: Task 293 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 784
parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/1e37b574-52b0-418c-a9db-600752e5554b_calibrate_855911_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/1e37b574-52b0-418c-a9db-600752e5554b_calibrate_855911_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/ef9a9b17-4054-4ff4-9f53-6d3528930de5_characterizeImage_855911_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/ef9a9b17-4054-4ff4-9f53-6d3528930de5_characterizeImage_855911_14.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:14 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 55 try 0 failed
parsl.dataflow.dflow INFO: Task 55 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/197d3d62-39c7-4303-b5c4-6f03c71107cb_characterizeImage_855911_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/197d3d62-39c7-4303-b5c4-6f03c71107cb_characterizeImage_855911_22.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 2ae285dffcef1e0e82b3bb35f8a3a3cf
parsl.dataflow.memoization INFO: Task 55 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 55 try 1 launched on executor multi with executor id 785
parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/197d3d62-39c7-4303-b5c4-6f03c71107cb_characterizeImage_855911_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/197d3d62-39c7-4303-b5c4-6f03c71107cb_characterizeImage_855911_22.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:14 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 1003 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 01652f8125662f1ed1978148f5f34d7b with result from task 1003
parsl.dataflow.dflow DEBUG: Task 1003 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1004 has memoization hash adf6cb0f74bf726d49569545af5a744c
parsl.dataflow.memoization INFO: Task 1004 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1004 try 0 launched on executor multi with executor id 786
parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/c0b60462-6b9e-4161-9fe1-300e9185d26d_calibrate_855911_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/c0b60462-6b9e-4161-9fe1-300e9185d26d_calibrate_855911_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/5e3181f6-b96b-42fe-bdd9-398afb2e2852_characterizeImage_855911_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/5e3181f6-b96b-42fe-bdd9-398afb2e2852_characterizeImage_855911_32.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 640 try 0 failed
parsl.dataflow.dflow INFO: Task 640 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/87fffcf9-43c1-4b40-8db9-3ecba65a35b5_characterizeImage_855910_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/87fffcf9-43c1-4b40-8db9-3ecba65a35b5_characterizeImage_855910_25.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 640 has memoization hash b82150ed2b6c8575218c021fafb8c554
parsl.dataflow.memoization INFO: Task 640 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 640 try 1 launched on executor multi with executor id 787
parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/87fffcf9-43c1-4b40-8db9-3ecba65a35b5_characterizeImage_855910_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/87fffcf9-43c1-4b40-8db9-3ecba65a35b5_characterizeImage_855910_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:15 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:15 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 679 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d15e453695bd70ab742b3a8d374355ed with result from task 679
parsl.dataflow.dflow DEBUG: Task 679 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 680 has memoization hash 835dbf2b34cefd178e6349701b0aac0f
parsl.dataflow.memoization INFO: Task 680 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15808e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 680 try 0 launched on executor multi with executor id 788
parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/014bfeba-e23c-440e-a439-302ceb85b836_calibrate_855911_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/014bfeba-e23c-440e-a439-302ceb85b836_calibrate_855911_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/366329cc-e8f6-4267-8900-06f4967cc695_characterizeImage_855911_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/366329cc-e8f6-4267-8900-06f4967cc695_characterizeImage_855911_54.stderr
parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 47554c015f0ad7bb8482c47850623974 with result from task 16
parsl.dataflow.dflow DEBUG: Task 16 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 2d86bb2538c7d94f4572378638481385
parsl.dataflow.memoization INFO: Task 17 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fdb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 789
parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/6feb35c8-6162-4b82-8dde-a8f20aef3a01_calibrate_855909_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/6feb35c8-6162-4b82-8dde-a8f20aef3a01_calibrate_855909_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/730f3db9-d95d-4bb9-b528-26acafd80720_characterizeImage_855909_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/730f3db9-d95d-4bb9-b528-26acafd80720_characterizeImage_855909_45.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 763 try 1 failed
parsl.dataflow.dflow ERROR: Task 763 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 0d6526c83493d3169a5daf6d48f0a3a9 with result from task 763
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 764 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 764 try 0 failed
parsl.dataflow.dflow INFO: Task 764 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/042cf385-602a-41cd-b3e5-a596f0c24608_calibrate_855821_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/042cf385-602a-41cd-b3e5-a596f0c24608_calibrate_855821_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/6fc25a6f-6e01-4365-b3d2-a1a5e48c5ed9_characterizeImage_855821_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/6fc25a6f-6e01-4365-b3d2-a1a5e48c5ed9_characterizeImage_855821_2.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:16 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 406 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ce9204ec298c50eec38aefd5606a1ce7 with result from task 406
parsl.dataflow.dflow DEBUG: Task 406 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 407 has memoization hash 17e6d9d94e8a714dfe6501687656f916
parsl.dataflow.memoization INFO: Task 407 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 790
parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/91250870-cd13-409c-9e8d-751891514cdd_calibrate_855911_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/91250870-cd13-409c-9e8d-751891514cdd_calibrate_855911_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/eb64cad1-af37-4541-872f-304a0c3b7ef7_characterizeImage_855911_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/eb64cad1-af37-4541-872f-304a0c3b7ef7_characterizeImage_855911_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:16 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0c7064d8d70118060abc4d39fafa6f00 with result from task 13
parsl.dataflow.dflow DEBUG: Task 13 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 8d5e876f730b3a02c3f653f54e1d311a
parsl.dataflow.memoization INFO: Task 14 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fd300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 791
parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/2fce3322-7467-48eb-902b-7ce876a1e6ba_calibrate_855911_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/2fce3322-7467-48eb-902b-7ce876a1e6ba_calibrate_855911_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/624cec9e-6cce-47d8-ad03-90deacaefc63_characterizeImage_855911_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/624cec9e-6cce-47d8-ad03-90deacaefc63_characterizeImage_855911_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:17 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 992d02b07d06394b7645335d1e50fe96 with result from task 343
parsl.dataflow.dflow DEBUG: Task 343 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 344 has memoization hash 5cf542e9a8ec8b771af6236ba238aeb9
parsl.dataflow.memoization INFO: Task 344 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16abae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 792
parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/a18929ed-8e8a-47e6-aeb0-e6b877227cde_calibrate_855910_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/a18929ed-8e8a-47e6-aeb0-e6b877227cde_calibrate_855910_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/1bcf117f-bad4-4fc6-8fd0-b2b31835728d_characterizeImage_855910_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/1bcf117f-bad4-4fc6-8fd0-b2b31835728d_characterizeImage_855910_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:18 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 339ad9c3326aedaa127a2800a4103eb2 with result from task 151
parsl.dataflow.dflow DEBUG: Task 151 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 79ef392d2714494703d5ab4beb319402
parsl.dataflow.memoization INFO: Task 152 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a425c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 793
parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/137f21d7-a61d-4a26-953c-bf98c691df7e_calibrate_855911_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/137f21d7-a61d-4a26-953c-bf98c691df7e_calibrate_855911_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/96a3abcb-659e-4f2d-94ac-7e1b4c5d3f86_characterizeImage_855911_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/96a3abcb-659e-4f2d-94ac-7e1b4c5d3f86_characterizeImage_855911_41.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:18 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 589 try 0 failed
parsl.dataflow.dflow INFO: Task 589 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/dbcece99-9605-4365-9c86-3edcd40507ab_characterizeImage_855911_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/dbcece99-9605-4365-9c86-3edcd40507ab_characterizeImage_855911_3.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 589 has memoization hash 98eade9ac5578605139a9f6ded2dba22
parsl.dataflow.memoization INFO: Task 589 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 589 try 1 launched on executor multi with executor id 794
parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/dbcece99-9605-4365-9c86-3edcd40507ab_characterizeImage_855911_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/dbcece99-9605-4365-9c86-3edcd40507ab_characterizeImage_855911_3.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 430 try 1 failed
parsl.dataflow.dflow ERROR: Task 430 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 3b27aea4bd6c4c814410fb1834776b53 with result from task 430
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 431 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 431 try 0 failed
parsl.dataflow.dflow INFO: Task 431 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/505943a8-0672-4485-a2a3-6205dd71da18_calibrate_855910_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/505943a8-0672-4485-a2a3-6205dd71da18_calibrate_855910_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7257da59-452a-496b-95ff-a7fe3d3cef53_characterizeImage_855910_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7257da59-452a-496b-95ff-a7fe3d3cef53_characterizeImage_855910_2.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:19 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 62f3d0459baa14ae7b6674ecf5ecc238 with result from task 145
parsl.dataflow.dflow DEBUG: Task 145 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 676b73370afe2f1ef4f39f09ee6e1c1b
parsl.dataflow.memoization INFO: Task 146 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a41f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 795
parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/d0b16eaa-7653-4e4e-8d2d-6b5e09af82b8_calibrate_855910_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/d0b16eaa-7653-4e4e-8d2d-6b5e09af82b8_calibrate_855910_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/2596cad3-9b26-4917-bb13-8b4035f77480_characterizeImage_855910_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/2596cad3-9b26-4917-bb13-8b4035f77480_characterizeImage_855910_34.stderr
parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 18a6e3ca9ec37edc660f4ffb6b2929be with result from task 232
parsl.dataflow.dflow DEBUG: Task 232 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 233 has memoization hash d8f4373dcdc37acbfdeff85366d9ac1e
parsl.dataflow.memoization INFO: Task 233 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a67a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:19 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 796
parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/1cf7050f-addf-495d-8b9c-922a03ecd6aa_calibrate_855909_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/1cf7050f-addf-495d-8b9c-922a03ecd6aa_calibrate_855909_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/77e5bdb6-dd7e-4086-aafd-23c417e7c246_characterizeImage_855909_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/77e5bdb6-dd7e-4086-aafd-23c417e7c246_characterizeImage_855909_26.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 712 try 0 failed
parsl.dataflow.dflow INFO: Task 712 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b8d5e5fe-2daf-48fe-949a-bea601c3e3df_characterizeImage_855910_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b8d5e5fe-2daf-48fe-949a-bea601c3e3df_characterizeImage_855910_13.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 712 has memoization hash d953d888a7e2164ad5898ef37639131d
parsl.dataflow.memoization INFO: Task 712 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 712 try 1 launched on executor multi with executor id 797
parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b8d5e5fe-2daf-48fe-949a-bea601c3e3df_characterizeImage_855910_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b8d5e5fe-2daf-48fe-949a-bea601c3e3df_characterizeImage_855910_13.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 616 try 0 failed
parsl.dataflow.dflow INFO: Task 616 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/314d9a2e-afb2-44b7-aae0-d60cc3469b3a_characterizeImage_855911_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/314d9a2e-afb2-44b7-aae0-d60cc3469b3a_characterizeImage_855911_17.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 616 has memoization hash e96d371890aa491733302d21a1b67712
parsl.dataflow.memoization INFO: Task 616 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 616 try 1 launched on executor multi with executor id 798
parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/314d9a2e-afb2-44b7-aae0-d60cc3469b3a_characterizeImage_855911_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/314d9a2e-afb2-44b7-aae0-d60cc3469b3a_characterizeImage_855911_17.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 571 try 0 failed
parsl.dataflow.dflow INFO: Task 571 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/d2050731-17bb-472a-a78e-a1f0abadb25f_characterizeImage_855911_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/d2050731-17bb-472a-a78e-a1f0abadb25f_characterizeImage_855911_39.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 571 has memoization hash cdac144bbb8353fc5cabbca7a2b0df76
parsl.dataflow.memoization INFO: Task 571 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 571 try 1 launched on executor multi with executor id 799
parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/d2050731-17bb-472a-a78e-a1f0abadb25f_characterizeImage_855911_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/d2050731-17bb-472a-a78e-a1f0abadb25f_characterizeImage_855911_39.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 208 try 1 failed
parsl.dataflow.dflow ERROR: Task 208 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 77b95601efaab5aa94e501a23b281932 with result from task 208
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 209 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 209 try 0 failed
parsl.dataflow.dflow INFO: Task 209 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/8a65a9c6-9f7a-410f-8f99-ba58c21caba1_calibrate_855911_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/8a65a9c6-9f7a-410f-8f99-ba58c21caba1_calibrate_855911_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/1772a9f8-c709-426c-ab7c-2153ab24dcfb_characterizeImage_855911_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/1772a9f8-c709-426c-ab7c-2153ab24dcfb_characterizeImage_855911_4.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:20 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 532 try 0 failed
parsl.dataflow.dflow INFO: Task 532 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b28ff9c2-f409-4140-8f81-68513963b66a_characterizeImage_855910_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b28ff9c2-f409-4140-8f81-68513963b66a_characterizeImage_855910_28.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 532 has memoization hash 7ae61edb24360387beb85e59a58531cc
parsl.dataflow.memoization INFO: Task 532 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15947600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 532 try 1 launched on executor multi with executor id 800
parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b28ff9c2-f409-4140-8f81-68513963b66a_characterizeImage_855910_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b28ff9c2-f409-4140-8f81-68513963b66a_characterizeImage_855910_28.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 331 try 1 failed
parsl.dataflow.dflow ERROR: Task 331 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 31c98fd7bcb29d807bc43f5178c741c7 with result from task 331
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 332 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 332 try 0 failed
parsl.dataflow.dflow INFO: Task 332 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/38b01187-cfd2-4d63-88a1-0f0ffe41e400_calibrate_855911_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/38b01187-cfd2-4d63-88a1-0f0ffe41e400_calibrate_855911_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/43ac2e76-0ffa-420b-952a-048e40aa8647_characterizeImage_855911_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/43ac2e76-0ffa-420b-952a-048e40aa8647_characterizeImage_855911_23.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 73 try 1 failed
parsl.dataflow.dflow ERROR: Task 73 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 250d620d1601e63d735774a717cd6274 with result from task 73
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 74 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 74 try 0 failed
parsl.dataflow.dflow INFO: Task 74 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/ad1b65db-52f6-4f26-8d55-65ed45fac32b_calibrate_855910_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/ad1b65db-52f6-4f26-8d55-65ed45fac32b_calibrate_855910_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/54458e78-9027-47ce-862b-4d889c61ef6f_characterizeImage_855910_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/54458e78-9027-47ce-862b-4d889c61ef6f_characterizeImage_855910_45.stderr
parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2023bebb8a8160e0d4f8e47a5fe99512 with result from task 271
parsl.dataflow.dflow DEBUG: Task 271 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 272 has memoization hash 3b916828894c37f12d7107835046a03f
parsl.dataflow.memoization INFO: Task 272 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 801
parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/aa1093dd-173c-4b83-bdde-3cc85224f5d1_calibrate_855909_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/aa1093dd-173c-4b83-bdde-3cc85224f5d1_calibrate_855909_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/3a71f63b-37d0-4ad6-babb-85e99033a7c5_characterizeImage_855909_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/3a71f63b-37d0-4ad6-babb-85e99033a7c5_characterizeImage_855909_7.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 187 try 1 failed
parsl.dataflow.dflow ERROR: Task 187 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry a7ee76acc3a5fd0fd879cd4fd7b422ab with result from task 187
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 188 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 188 try 0 failed
parsl.dataflow.dflow INFO: Task 188 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/f6cce5a6-c04b-4281-affb-ef7fcabdb4cd_calibrate_855911_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/f6cce5a6-c04b-4281-affb-ef7fcabdb4cd_calibrate_855911_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/380f55c6-32ea-4c0c-96bb-f30cf90e608d_characterizeImage_855911_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/380f55c6-32ea-4c0c-96bb-f30cf90e608d_characterizeImage_855911_29.stderr
parsl.dataflow.dflow INFO: Task 415 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 59e089908d170d746641af59df492171 with result from task 415
parsl.dataflow.dflow DEBUG: Task 415 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 416 has memoization hash ad62d2061ea67f2b8978ddf93360b5a7
parsl.dataflow.memoization INFO: Task 416 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 416 try 0 launched on executor multi with executor id 802
parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/e5ef701f-3613-4c56-bad4-5d3becd57b80_calibrate_855911_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/e5ef701f-3613-4c56-bad4-5d3becd57b80_calibrate_855911_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/060e94fe-621c-44b6-8142-678759c1f991_characterizeImage_855911_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/060e94fe-621c-44b6-8142-678759c1f991_characterizeImage_855911_7.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:20 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 802 try 0 failed
parsl.dataflow.dflow INFO: Task 802 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a3f165c3-e409-4667-a596-123d9ae5bbee_characterizeImage_855910_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a3f165c3-e409-4667-a596-123d9ae5bbee_characterizeImage_855910_39.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 802 has memoization hash 31c89347931d428ba7178e70dbd3a85a
parsl.dataflow.memoization INFO: Task 802 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15895080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 802 try 1 launched on executor multi with executor id 803
parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a3f165c3-e409-4667-a596-123d9ae5bbee_characterizeImage_855910_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a3f165c3-e409-4667-a596-123d9ae5bbee_characterizeImage_855910_39.stderr
parsl.dataflow.dflow INFO: Task 637 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4962ac62aa8a84962ab2f433f0ed4efd with result from task 637
parsl.dataflow.dflow DEBUG: Task 637 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 638 has memoization hash f5dbba5a5cfbf9d3db4fdeb5a408a8ba
parsl.dataflow.memoization INFO: Task 638 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 638 try 0 launched on executor multi with executor id 804
parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/db9572b5-8e4e-4a25-a2f9-063ffcddfbde_calibrate_855911_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/db9572b5-8e4e-4a25-a2f9-063ffcddfbde_calibrate_855911_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/fba0a16e-9a93-448b-a2f6-9940048ff6d5_characterizeImage_855911_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/fba0a16e-9a93-448b-a2f6-9940048ff6d5_characterizeImage_855911_49.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:21 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e46a7a47b5b5f033e660f14b23907856 with result from task 154
parsl.dataflow.dflow DEBUG: Task 154 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 533d28244adb048d0a6ae4cfa5041b14
parsl.dataflow.memoization INFO: Task 155 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a428e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 805
parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/608b3a90-8ab2-4965-a13e-3da6418aaf24_calibrate_855911_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/608b3a90-8ab2-4965-a13e-3da6418aaf24_calibrate_855911_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/7d74dfe9-174b-4646-a32d-222ccb910e73_characterizeImage_855911_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/7d74dfe9-174b-4646-a32d-222ccb910e73_characterizeImage_855911_37.stderr
parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 85cb0c51440dcd0df1aa0359cc43cde0 with result from task 274
parsl.dataflow.dflow DEBUG: Task 274 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 7f32b4c06bbf3e95f398e95d322f12dd
parsl.dataflow.memoization INFO: Task 275 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 806
parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/b963d7b7-0577-462a-a0b8-16262e565d94_calibrate_855910_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/b963d7b7-0577-462a-a0b8-16262e565d94_calibrate_855910_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/6df5a330-f7f2-449a-9a11-a744dffbcd6a_characterizeImage_855910_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/6df5a330-f7f2-449a-9a11-a744dffbcd6a_characterizeImage_855910_52.stderr
parsl.dataflow.dflow INFO: Task 409 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3338ff462c27bf0bbc30271fafdfa6c9 with result from task 409
parsl.dataflow.dflow DEBUG: Task 409 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 410 has memoization hash d4c2b9e9671296d14a2959b36d7e29e1
parsl.dataflow.memoization INFO: Task 410 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 410 try 0 launched on executor multi with executor id 807
parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/c5e764df-5046-4c12-8e66-2a74ab581427_calibrate_855911_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/c5e764df-5046-4c12-8e66-2a74ab581427_calibrate_855911_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/24f7d608-6781-4af4-b04b-1ac9318bdaee_characterizeImage_855911_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/24f7d608-6781-4af4-b04b-1ac9318bdaee_characterizeImage_855911_42.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 709 try 0 failed
parsl.dataflow.dflow INFO: Task 709 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/41f4abe5-116f-4461-bb35-e4782aed1809_characterizeImage_855910_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/41f4abe5-116f-4461-bb35-e4782aed1809_characterizeImage_855910_40.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 709 has memoization hash 37005d6a6d1cd9fb6effc35f1fc4054d
parsl.dataflow.memoization INFO: Task 709 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 709 try 1 launched on executor multi with executor id 808
parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/41f4abe5-116f-4461-bb35-e4782aed1809_characterizeImage_855910_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/41f4abe5-116f-4461-bb35-e4782aed1809_characterizeImage_855910_40.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:22 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 328 try 1 failed
parsl.dataflow.dflow ERROR: Task 328 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 32413be48fa0fc7becb3f0064b213c09 with result from task 328
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 329 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 329 try 0 failed
parsl.dataflow.dflow INFO: Task 329 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/30433400-9df7-430b-89c6-bb55e0dc4bfc_calibrate_855911_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/30433400-9df7-430b-89c6-bb55e0dc4bfc_calibrate_855911_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/9b622d83-8f4e-40ef-a3db-f40da51be251_characterizeImage_855911_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/9b622d83-8f4e-40ef-a3db-f40da51be251_characterizeImage_855911_31.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 199 try 1 failed
parsl.dataflow.dflow ERROR: Task 199 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry a807ccd39a385f95f710aec7ff05a570 with result from task 199
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 200 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 200 try 0 failed
parsl.dataflow.dflow INFO: Task 200 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/76e2c18a-750e-4e78-a2a5-598b381ce0a2_calibrate_855910_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/76e2c18a-750e-4e78-a2a5-598b381ce0a2_calibrate_855910_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/ac9d00eb-08f8-4dba-acad-e1bb1800f245_characterizeImage_855910_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/ac9d00eb-08f8-4dba-acad-e1bb1800f245_characterizeImage_855910_15.stderr
parsl.dataflow.dflow INFO: Task 499 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry de6438478a8e43846cce1fba58d32119 with result from task 499
parsl.dataflow.dflow DEBUG: Task 499 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 500 has memoization hash c41c62e11b12a82d284db429b68a5141
parsl.dataflow.memoization INFO: Task 500 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159451c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 500 try 0 launched on executor multi with executor id 809
parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/27548139-64d6-467b-b22c-bf3d8a966312_calibrate_855909_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/27548139-64d6-467b-b22c-bf3d8a966312_calibrate_855909_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/a0381633-859b-4399-b7aa-962f9fe790f8_characterizeImage_855909_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/a0381633-859b-4399-b7aa-962f9fe790f8_characterizeImage_855909_20.stderr
parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b28f369b3faea4b21e8e1be6c45c0cd2 with result from task 253
parsl.dataflow.dflow DEBUG: Task 253 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 254 has memoization hash 09653f709a778a4809b1365d959fef09
parsl.dataflow.memoization INFO: Task 254 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 810
parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/ed8b204f-f9e0-4fae-8e2a-5436dc4d5298_calibrate_855910_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/ed8b204f-f9e0-4fae-8e2a-5436dc4d5298_calibrate_855910_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9b4ba870-7e1b-465c-a0ab-7fa4bb4af0c8_characterizeImage_855910_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9b4ba870-7e1b-465c-a0ab-7fa4bb4af0c8_characterizeImage_855910_7.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 694 try 0 failed
parsl.dataflow.dflow INFO: Task 694 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/99231e1d-b415-4de5-8510-c098fb36585f_characterizeImage_855911_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/99231e1d-b415-4de5-8510-c098fb36585f_characterizeImage_855911_44.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 694 has memoization hash e028cd437b0013743831baa511937486
parsl.dataflow.memoization INFO: Task 694 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15809f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 694 try 1 launched on executor multi with executor id 811
parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/99231e1d-b415-4de5-8510-c098fb36585f_characterizeImage_855911_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/99231e1d-b415-4de5-8510-c098fb36585f_characterizeImage_855911_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:22 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.dataflow.dflow INFO: Task 790 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6174d350751e8537129f680a7b25bb79 with result from task 790
parsl.dataflow.dflow DEBUG: Task 790 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 791 has memoization hash 29dedd432a0174a19795feb0a91e7036
parsl.dataflow.memoization INFO: Task 791 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15894220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 791 try 0 launched on executor multi with executor id 812
parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/af04875e-68da-4810-8e8a-104f32db4360_calibrate_855910_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/af04875e-68da-4810-8e8a-104f32db4360_calibrate_855910_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/cf3d7b4a-b9c9-4c50-b5fd-d3626fc8f9be_characterizeImage_855910_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/cf3d7b4a-b9c9-4c50-b5fd-d3626fc8f9be_characterizeImage_855910_53.stderr
parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 41d70da2e315fa561af6a6a6f51b6e77 with result from task 295
parsl.dataflow.dflow DEBUG: Task 295 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 296 has memoization hash bf4c74e444a31c5685442b0b8b56c40f
parsl.dataflow.memoization INFO: Task 296 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 813
parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6911759e-2be1-4ef1-9c82-9b802e45f512_calibrate_855910_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6911759e-2be1-4ef1-9c82-9b802e45f512_calibrate_855910_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/0dd15248-135f-4cfb-8767-9155a894800d_characterizeImage_855910_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/0dd15248-135f-4cfb-8767-9155a894800d_characterizeImage_855910_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:23 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 307 try 1 failed
parsl.dataflow.dflow ERROR: Task 307 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry b638ea1d813b2bd661d044d5f67a3d62 with result from task 307
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 308 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 308 try 0 failed
parsl.dataflow.dflow INFO: Task 308 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/1653b55c-b4e9-4d94-906a-b0787b326c5b_calibrate_855910_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/1653b55c-b4e9-4d94-906a-b0787b326c5b_calibrate_855910_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9ed1bf3e-ea29-498b-8e9b-247ecfad18e0_characterizeImage_855910_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9ed1bf3e-ea29-498b-8e9b-247ecfad18e0_characterizeImage_855910_47.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 205 try 1 failed
parsl.dataflow.dflow ERROR: Task 205 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry c4da951aec1cde9c135aa677eacde2e7 with result from task 205
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 206 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 206 try 0 failed
parsl.dataflow.dflow INFO: Task 206 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/55e85659-ac40-4551-9c70-f1a12b9c1675_calibrate_855910_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/55e85659-ac40-4551-9c70-f1a12b9c1675_calibrate_855910_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a5fefbfe-5e7b-4e1a-976e-347e18a8306b_characterizeImage_855910_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a5fefbfe-5e7b-4e1a-976e-347e18a8306b_characterizeImage_855910_49.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 724 try 0 failed
parsl.dataflow.dflow INFO: Task 724 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/86f749f5-3fbf-4fb8-be29-20a0377b7713_characterizeImage_855911_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/86f749f5-3fbf-4fb8-be29-20a0377b7713_characterizeImage_855911_6.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 724 has memoization hash 539a1468b63de63caaefd742a0154733
parsl.dataflow.memoization INFO: Task 724 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 724 try 1 launched on executor multi with executor id 814
parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/86f749f5-3fbf-4fb8-be29-20a0377b7713_characterizeImage_855911_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/86f749f5-3fbf-4fb8-be29-20a0377b7713_characterizeImage_855911_6.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 1042 try 0 failed
parsl.dataflow.dflow INFO: Task 1042 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/7570243c-6ca2-44db-8516-bd1db9b982f4_characterizeImage_855911_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/7570243c-6ca2-44db-8516-bd1db9b982f4_characterizeImage_855911_45.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1042 has memoization hash c383d405c4b25e05360df755ee7af784
parsl.dataflow.memoization INFO: Task 1042 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1042 try 1 launched on executor multi with executor id 815
parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/7570243c-6ca2-44db-8516-bd1db9b982f4_characterizeImage_855911_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/7570243c-6ca2-44db-8516-bd1db9b982f4_characterizeImage_855911_45.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 364 try 1 failed
parsl.dataflow.dflow ERROR: Task 364 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry f4c0a937b2c8a4a5092a6b0aebf15ce2 with result from task 364
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 365 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 365 try 0 failed
parsl.dataflow.dflow INFO: Task 365 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/2d2a1483-82fa-496c-8498-6683ea4d283b_calibrate_855911_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/2d2a1483-82fa-496c-8498-6683ea4d283b_calibrate_855911_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/0488fa05-2443-4276-8395-c08857e44fb6_characterizeImage_855911_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/0488fa05-2443-4276-8395-c08857e44fb6_characterizeImage_855911_16.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 751 try 0 failed
parsl.dataflow.dflow INFO: Task 751 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/da0d1552-f18a-4d65-b727-c59d69ca334a_characterizeImage_855911_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/da0d1552-f18a-4d65-b727-c59d69ca334a_characterizeImage_855911_46.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 751 has memoization hash 3014026d02ff6831ef5034427d216632
parsl.dataflow.memoization INFO: Task 751 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15859b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 751 try 1 launched on executor multi with executor id 816
parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/da0d1552-f18a-4d65-b727-c59d69ca334a_characterizeImage_855911_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/da0d1552-f18a-4d65-b727-c59d69ca334a_characterizeImage_855911_46.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 355 try 1 failed
parsl.dataflow.dflow ERROR: Task 355 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 8c591d801a49ded05e4053bf75bb6bff with result from task 355
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 356 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 356 try 0 failed
parsl.dataflow.dflow INFO: Task 356 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/50ff2a93-074f-4c97-994c-99281cd74891_calibrate_855911_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/50ff2a93-074f-4c97-994c-99281cd74891_calibrate_855911_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/ab0f61a5-3ad2-4dec-8705-232145f6b45e_characterizeImage_855911_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/ab0f61a5-3ad2-4dec-8705-232145f6b45e_characterizeImage_855911_1.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 670 try 0 failed
parsl.dataflow.dflow INFO: Task 670 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/e25fe73f-595a-4f0f-8204-3dc027466c20_characterizeImage_855910_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/e25fe73f-595a-4f0f-8204-3dc027466c20_characterizeImage_855910_19.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 670 has memoization hash 35dde2a4d9ccd4f23e5e0dfc62840a90
parsl.dataflow.memoization INFO: Task 670 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15808680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 670 try 1 launched on executor multi with executor id 817
parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/e25fe73f-595a-4f0f-8204-3dc027466c20_characterizeImage_855910_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/e25fe73f-595a-4f0f-8204-3dc027466c20_characterizeImage_855910_19.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 970 try 1 failed
parsl.dataflow.dflow ERROR: Task 970 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry e49f48cdbbe775f891a411364cffb819 with result from task 970
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 971 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 971 try 0 failed
parsl.dataflow.dflow INFO: Task 971 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/ddbdeace-118c-4c87-9592-504f37ce0eba_calibrate_855911_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/ddbdeace-118c-4c87-9592-504f37ce0eba_calibrate_855911_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8ed5ca7a-d316-45a9-9282-658561e3897d_characterizeImage_855911_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8ed5ca7a-d316-45a9-9282-658561e3897d_characterizeImage_855911_48.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 541 try 0 failed
parsl.dataflow.dflow INFO: Task 541 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/2f717859-0942-4d79-b214-59d39385fa48_characterizeImage_855911_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/2f717859-0942-4d79-b214-59d39385fa48_characterizeImage_855911_53.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 541 has memoization hash 2afeeab34b1fa3be3465709379808987
parsl.dataflow.memoization INFO: Task 541 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15947f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 541 try 1 launched on executor multi with executor id 818
parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/2f717859-0942-4d79-b214-59d39385fa48_characterizeImage_855911_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/2f717859-0942-4d79-b214-59d39385fa48_characterizeImage_855911_53.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 664 try 0 failed
parsl.dataflow.dflow INFO: Task 664 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/630877c9-a022-4ed5-97ac-e679b28da27f_characterizeImage_855910_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/630877c9-a022-4ed5-97ac-e679b28da27f_characterizeImage_855910_46.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 664 has memoization hash 983270ae2cf3bab3fabbfa493f7cfb76
parsl.dataflow.memoization INFO: Task 664 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 664 try 1 launched on executor multi with executor id 819
parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/630877c9-a022-4ed5-97ac-e679b28da27f_characterizeImage_855910_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/630877c9-a022-4ed5-97ac-e679b28da27f_characterizeImage_855910_46.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 520 try 0 failed
parsl.dataflow.dflow INFO: Task 520 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/01c702af-c157-42c4-9e9d-62292b95238f_characterizeImage_855911_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/01c702af-c157-42c4-9e9d-62292b95238f_characterizeImage_855911_11.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 520 has memoization hash af4f7ecce07b87824f97859da083cecb
parsl.dataflow.memoization INFO: Task 520 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15946980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 520 try 1 launched on executor multi with executor id 820
parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/01c702af-c157-42c4-9e9d-62292b95238f_characterizeImage_855911_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/01c702af-c157-42c4-9e9d-62292b95238f_characterizeImage_855911_11.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 313 try 1 failed
parsl.dataflow.dflow ERROR: Task 313 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry a73354e138dcf9639491de69a740b98c with result from task 313
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 314 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 314 try 0 failed
parsl.dataflow.dflow INFO: Task 314 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/565e6444-2319-42f1-9086-6b9bd5e1a6eb_calibrate_855911_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/565e6444-2319-42f1-9086-6b9bd5e1a6eb_calibrate_855911_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8f7cdbb3-2180-4a76-a27b-d6bf52a1eebd_characterizeImage_855911_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8f7cdbb3-2180-4a76-a27b-d6bf52a1eebd_characterizeImage_855911_15.stderr
parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 66f83491c8abdfa9ce0e9ae385d8654a with result from task 109
parsl.dataflow.dflow DEBUG: Task 109 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 110 has memoization hash b0ae8687c4f7d60c8d45862cd8bc6fec
parsl.dataflow.memoization INFO: Task 110 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 821
parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/d398a040-72fb-4773-b23d-96bf076ea2c2_calibrate_855911_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/d398a040-72fb-4773-b23d-96bf076ea2c2_calibrate_855911_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/d62fb751-c5bb-4036-b9e0-ba333f7ef680_characterizeImage_855911_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/d62fb751-c5bb-4036-b9e0-ba333f7ef680_characterizeImage_855911_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:25 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 514 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 805a8f49a657086a45be339e7a2f4841 with result from task 514
parsl.dataflow.dflow DEBUG: Task 514 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 515 has memoization hash cf3a50475e3a0a8129a1b2f655dee89e
parsl.dataflow.memoization INFO: Task 515 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15946160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 515 try 0 launched on executor multi with executor id 822
parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/2376b548-b2c9-4ac2-8fd9-7db24fc33abb_calibrate_855910_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/2376b548-b2c9-4ac2-8fd9-7db24fc33abb_calibrate_855910_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/4c8d2724-36b6-4eb4-90d1-22de9d9cfed3_characterizeImage_855910_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/4c8d2724-36b6-4eb4-90d1-22de9d9cfed3_characterizeImage_855910_11.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 67 try 1 failed
parsl.dataflow.dflow ERROR: Task 67 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 72dafb5d26fd924d15ba7296be976376 with result from task 67
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 68 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 68 try 0 failed
parsl.dataflow.dflow INFO: Task 68 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/80808d56-a5b5-4f82-8340-fc99679bc4ec_calibrate_855909_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/80808d56-a5b5-4f82-8340-fc99679bc4ec_calibrate_855909_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/e48d4589-e0c5-4812-8dcb-94e46c3bd2fa_characterizeImage_855909_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/e48d4589-e0c5-4812-8dcb-94e46c3bd2fa_characterizeImage_855909_2.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 76 try 1 failed
parsl.dataflow.dflow ERROR: Task 76 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 6d86c1738ee5ffe58ba6fffb442a5fea with result from task 76
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 77 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 77 try 0 failed
parsl.dataflow.dflow INFO: Task 77 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/4c2161c4-9a1a-495a-a3f5-2f8b854975ab_calibrate_855822_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/4c2161c4-9a1a-495a-a3f5-2f8b854975ab_calibrate_855822_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a7ba73de-60e6-4bf8-a7dd-e884c059c8dd_characterizeImage_855822_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a7ba73de-60e6-4bf8-a7dd-e884c059c8dd_characterizeImage_855822_2.stderr
parsl.dataflow.dflow INFO: Task 388 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3780e9a62780018fc84e2f249e0c7966 with result from task 388
parsl.dataflow.dflow DEBUG: Task 388 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 389 has memoization hash 2572e5fb14491cb6acdb24c132f79f8f
parsl.dataflow.memoization INFO: Task 389 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 389 try 0 launched on executor multi with executor id 823
parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/f4305e01-5bd9-4b9d-9ae6-dfa67182e148_calibrate_855909_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/f4305e01-5bd9-4b9d-9ae6-dfa67182e148_calibrate_855909_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/6b2f241f-1025-4212-9ddb-6e251f4d1991_characterizeImage_855909_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/6b2f241f-1025-4212-9ddb-6e251f4d1991_characterizeImage_855909_46.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 721 try 0 failed
parsl.dataflow.dflow INFO: Task 721 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/cdb1cee3-7fa1-4ebc-bcf0-a9ef1ae1cfec_characterizeImage_855911_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/cdb1cee3-7fa1-4ebc-bcf0-a9ef1ae1cfec_characterizeImage_855911_13.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 721 has memoization hash 900a9cada33524e4e1d7252a074c1b34
parsl.dataflow.memoization INFO: Task 721 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 721 try 1 launched on executor multi with executor id 824
parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/cdb1cee3-7fa1-4ebc-bcf0-a9ef1ae1cfec_characterizeImage_855911_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/cdb1cee3-7fa1-4ebc-bcf0-a9ef1ae1cfec_characterizeImage_855911_13.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 4 try 1 failed
parsl.dataflow.dflow ERROR: Task 4 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 9eec1aeaf65e20888775993fac73ff1f with result from task 4
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 5 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 5 try 0 failed
parsl.dataflow.dflow INFO: Task 5 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/b2785e32-e721-4193-be7c-ad9474845bef_calibrate_855911_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/b2785e32-e721-4193-be7c-ad9474845bef_calibrate_855911_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/3876bf01-95f0-4738-8e5d-a3052cf869b8_characterizeImage_855911_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/3876bf01-95f0-4738-8e5d-a3052cf869b8_characterizeImage_855911_33.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:26 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 373 try 1 failed
parsl.dataflow.dflow ERROR: Task 373 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry cb69a8e81091c970c23065e7690154af with result from task 373
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 374 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 374 try 0 failed
parsl.dataflow.dflow INFO: Task 374 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/0241e401-66b2-4437-a300-8f878057a290_calibrate_855910_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/0241e401-66b2-4437-a300-8f878057a290_calibrate_855910_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/5f20bc73-308d-49d7-b631-f3754c89a722_characterizeImage_855910_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/5f20bc73-308d-49d7-b631-f3754c89a722_characterizeImage_855910_32.stderr
parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 28bf0f55dd02f7b245dc5874e8169997 with result from task 34
parsl.dataflow.dflow DEBUG: Task 34 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 35 has memoization hash 6e8fabcdc812273db85f53adcc588782
parsl.dataflow.memoization INFO: Task 35 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fe7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 825
parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/f9e3a686-e7de-482f-be9b-3537a6bb7fed_calibrate_855910_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/f9e3a686-e7de-482f-be9b-3537a6bb7fed_calibrate_855910_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/5d3158c4-6eb6-4ef8-b1ee-c628bc43e0c5_characterizeImage_855910_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/5d3158c4-6eb6-4ef8-b1ee-c628bc43e0c5_characterizeImage_855910_56.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 511 try 1 failed
parsl.dataflow.dflow ERROR: Task 511 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry c8f242c7b9e33d0220c71832797d4823 with result from task 511
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 512 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 512 try 0 failed
parsl.dataflow.dflow INFO: Task 512 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/12c9e395-ac2d-4f29-96ba-173d95b84780_calibrate_855911_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/12c9e395-ac2d-4f29-96ba-173d95b84780_calibrate_855911_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/0fa21b17-4bee-4749-be6f-82640d5b659f_characterizeImage_855911_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/0fa21b17-4bee-4749-be6f-82640d5b659f_characterizeImage_855911_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:27 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 649 try 1 failed
parsl.dataflow.dflow ERROR: Task 649 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 6c16083a6bc22a33f459496b5e9d982d with result from task 649
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 650 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 650 try 0 failed
parsl.dataflow.dflow INFO: Task 650 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/3f547c64-f173-470b-a956-c39ff2d69a43_calibrate_855911_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/3f547c64-f173-470b-a956-c39ff2d69a43_calibrate_855911_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/b86f8dcf-7ba5-4429-b66e-65fba80f6930_characterizeImage_855911_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/b86f8dcf-7ba5-4429-b66e-65fba80f6930_characterizeImage_855911_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:28 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 943 try 0 failed
parsl.dataflow.dflow INFO: Task 943 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c2cb5987-4744-4a73-a11e-01711df145b9_characterizeImage_855911_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c2cb5987-4744-4a73-a11e-01711df145b9_characterizeImage_855911_8.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 943 has memoization hash 58ecb3603abbc2064411e53d0e773887
parsl.dataflow.memoization INFO: Task 943 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 943 try 1 launched on executor multi with executor id 826
parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c2cb5987-4744-4a73-a11e-01711df145b9_characterizeImage_855911_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c2cb5987-4744-4a73-a11e-01711df145b9_characterizeImage_855911_8.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 370 try 1 failed
parsl.dataflow.dflow ERROR: Task 370 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry c01a7dbf4f56fa06026a82b866dccf1d with result from task 370
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 371 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 371 try 0 failed
parsl.dataflow.dflow INFO: Task 371 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/da1aa9ed-7349-4ded-bd73-d820399abe74_calibrate_855911_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/da1aa9ed-7349-4ded-bd73-d820399abe74_calibrate_855911_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/17d4071f-0f92-4732-8a33-e6d45e1b118d_characterizeImage_855911_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/17d4071f-0f92-4732-8a33-e6d45e1b118d_characterizeImage_855911_19.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 1072 try 0 failed
parsl.dataflow.dflow INFO: Task 1072 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a0902891-8e8e-4426-96ef-365db6e697c5_characterizeImage_855910_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a0902891-8e8e-4426-96ef-365db6e697c5_characterizeImage_855910_26.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1072 has memoization hash 4d110a22fd0f938163de43e6f2161390
parsl.dataflow.memoization INFO: Task 1072 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1072 try 1 launched on executor multi with executor id 827
parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a0902891-8e8e-4426-96ef-365db6e697c5_characterizeImage_855910_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a0902891-8e8e-4426-96ef-365db6e697c5_characterizeImage_855910_26.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 940 try 0 failed
parsl.dataflow.dflow INFO: Task 940 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/9aa1ef21-e968-49df-89ad-1ceccc2ff409_characterizeImage_855911_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/9aa1ef21-e968-49df-89ad-1ceccc2ff409_characterizeImage_855911_47.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 940 has memoization hash 31dfc87fecdbb2ffc6efb50c17cd23b4
parsl.dataflow.memoization INFO: Task 940 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 940 try 1 launched on executor multi with executor id 828
parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/9aa1ef21-e968-49df-89ad-1ceccc2ff409_characterizeImage_855911_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/9aa1ef21-e968-49df-89ad-1ceccc2ff409_characterizeImage_855911_47.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:28 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 823 try 0 failed
parsl.dataflow.dflow INFO: Task 823 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8af071b6-eb9b-4d32-82ed-ce59eb90ebc7_characterizeImage_855911_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8af071b6-eb9b-4d32-82ed-ce59eb90ebc7_characterizeImage_855911_20.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 823 has memoization hash f14718381899a1619e1cf3f09a95ef33
parsl.dataflow.memoization INFO: Task 823 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15896660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 823 try 1 launched on executor multi with executor id 829
parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8af071b6-eb9b-4d32-82ed-ce59eb90ebc7_characterizeImage_855911_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8af071b6-eb9b-4d32-82ed-ce59eb90ebc7_characterizeImage_855911_20.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 382 try 1 failed
parsl.dataflow.dflow ERROR: Task 382 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry cf3065ae5f0e177351fb8bb10e157752 with result from task 382
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 383 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 383 try 0 failed
parsl.dataflow.dflow INFO: Task 383 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/9c5a49ff-6c7b-43ee-9728-20785f794ef0_calibrate_855910_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/9c5a49ff-6c7b-43ee-9728-20785f794ef0_calibrate_855910_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9385d400-89ab-4c09-9287-eb72b6bb36a3_characterizeImage_855910_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9385d400-89ab-4c09-9287-eb72b6bb36a3_characterizeImage_855910_33.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 28 try 1 failed
parsl.dataflow.dflow ERROR: Task 28 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry d65319fdaf240d1fd266d4fdc7251656 with result from task 28
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 29 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 29 try 0 failed
parsl.dataflow.dflow INFO: Task 29 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/b8279ffa-a050-4bb0-a5c7-7a0523d1c704_calibrate_855910_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/b8279ffa-a050-4bb0-a5c7-7a0523d1c704_calibrate_855910_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/26eb6126-3dec-4e32-89ba-2f15a60704bb_characterizeImage_855910_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/26eb6126-3dec-4e32-89ba-2f15a60704bb_characterizeImage_855910_1.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 103 try 1 failed
parsl.dataflow.dflow ERROR: Task 103 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 36686cc28c22241ddbd9aaa70188db58 with result from task 103
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 104 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 104 try 0 failed
parsl.dataflow.dflow INFO: Task 104 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/d2f9a810-731d-412e-9d2f-1dc15c545df8_calibrate_855911_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/d2f9a810-731d-412e-9d2f-1dc15c545df8_calibrate_855911_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/f321db45-b3e0-4ce4-a456-38b579e7c1ff_characterizeImage_855911_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/f321db45-b3e0-4ce4-a456-38b579e7c1ff_characterizeImage_855911_38.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:29 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 346 try 1 failed
parsl.dataflow.dflow ERROR: Task 346 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 0c1a81737dba7926fed2c9acca6004e9 with result from task 346
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 347 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 347 try 0 failed
parsl.dataflow.dflow INFO: Task 347 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/9c3504f9-6823-4df0-ac04-045a9d8c1e1f_calibrate_855910_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/9c3504f9-6823-4df0-ac04-045a9d8c1e1f_calibrate_855910_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7e0bf724-a995-4352-b9f6-334d12aed276_characterizeImage_855910_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7e0bf724-a995-4352-b9f6-334d12aed276_characterizeImage_855910_27.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 577 try 1 failed
parsl.dataflow.dflow ERROR: Task 577 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry d72d77aeba683cb553260f47f0ca9d24 with result from task 577
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 578 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 578 try 0 failed
parsl.dataflow.dflow INFO: Task 578 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6d52947b-daa1-4f3c-91e8-6cdffe168377_calibrate_855910_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6d52947b-daa1-4f3c-91e8-6cdffe168377_calibrate_855910_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b0af1003-e9b5-405e-9c1e-712df5a6a216_characterizeImage_855910_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b0af1003-e9b5-405e-9c1e-712df5a6a216_characterizeImage_855910_16.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 913 try 0 failed
parsl.dataflow.dflow INFO: Task 913 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/cb8d9282-f6ee-47b2-b580-a53006f2c1ce_characterizeImage_855910_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/cb8d9282-f6ee-47b2-b580-a53006f2c1ce_characterizeImage_855910_51.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 913 has memoization hash 0019dd4534ec95365f8fdd85ca1abd1b
parsl.dataflow.memoization INFO: Task 913 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157284a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 913 try 1 launched on executor multi with executor id 830
parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/cb8d9282-f6ee-47b2-b580-a53006f2c1ce_characterizeImage_855910_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/cb8d9282-f6ee-47b2-b580-a53006f2c1ce_characterizeImage_855910_51.stderr
parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 88f36cd4e3ce63fd6ff9070d5ea2e6fb with result from task 94
parsl.dataflow.dflow DEBUG: Task 94 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 5c95d1101c9012259ef8b60daf5c016c
parsl.dataflow.memoization INFO: Task 95 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 831
parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/9209a414-c040-4101-9a58-82fd120b8de4_calibrate_855910_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/9209a414-c040-4101-9a58-82fd120b8de4_calibrate_855910_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/6af7edc2-50de-4381-9d68-e22864275a80_characterizeImage_855910_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/6af7edc2-50de-4381-9d68-e22864275a80_characterizeImage_855910_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:29 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 676 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 451812b1dab00a469a518791c02c0260 with result from task 676
parsl.dataflow.dflow DEBUG: Task 676 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 677 has memoization hash 8c9fb1ff44683cb100b5b6f724b758bf
parsl.dataflow.memoization INFO: Task 677 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15808ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 677 try 0 launched on executor multi with executor id 832
parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/4799e5c2-0f26-4c4f-b5cf-7a334fd6e5e3_calibrate_855911_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/4799e5c2-0f26-4c4f-b5cf-7a334fd6e5e3_calibrate_855911_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/5eecd24c-477b-4bef-b2db-2e7313305e4e_characterizeImage_855911_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/5eecd24c-477b-4bef-b2db-2e7313305e4e_characterizeImage_855911_62.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 847 try 0 failed
parsl.dataflow.dflow INFO: Task 847 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/f85e8d7d-c248-499a-aa41-99ed2a946c91_characterizeImage_855910_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/f85e8d7d-c248-499a-aa41-99ed2a946c91_characterizeImage_855910_5.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 847 has memoization hash 8242a98cac56c33208355dd500abf3df
parsl.dataflow.memoization INFO: Task 847 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15897f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 847 try 1 launched on executor multi with executor id 833
parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/f85e8d7d-c248-499a-aa41-99ed2a946c91_characterizeImage_855910_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/f85e8d7d-c248-499a-aa41-99ed2a946c91_characterizeImage_855910_5.stderr
parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7cfdb8d27b1724859b2add5dc7bbc51b with result from task 91
parsl.dataflow.dflow DEBUG: Task 91 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 019018ca9c18eeec61b7fb7feb9bdc53
parsl.dataflow.memoization INFO: Task 92 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 834
parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/82fd2d5a-8b6a-4e79-aa92-6b90c754d4c2_calibrate_855910_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/82fd2d5a-8b6a-4e79-aa92-6b90c754d4c2_calibrate_855910_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/72a0c7e7-fa1a-4a3f-b3a4-ae297767711c_characterizeImage_855910_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/72a0c7e7-fa1a-4a3f-b3a4-ae297767711c_characterizeImage_855910_37.stderr
parsl.dataflow.dflow INFO: Task 646 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 90879420c427ddffa54bc42d8b0c7e34 with result from task 646
parsl.dataflow.dflow DEBUG: Task 646 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 647 has memoization hash 7474f37e89b63bbfb6f47ac580398836
parsl.dataflow.memoization INFO: Task 647 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 647 try 0 launched on executor multi with executor id 835
parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/6ded6f27-f613-4785-8a8d-9a63d58105b1_calibrate_855909_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/6ded6f27-f613-4785-8a8d-9a63d58105b1_calibrate_855909_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/7e2824a9-eef9-4c96-8626-11c7622aedd3_characterizeImage_855909_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/7e2824a9-eef9-4c96-8626-11c7622aedd3_characterizeImage_855909_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:30 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:31 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 928 try 0 failed
parsl.dataflow.dflow INFO: Task 928 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/539fcb88-d75b-4b6e-8708-537ee6010854_characterizeImage_855911_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/539fcb88-d75b-4b6e-8708-537ee6010854_characterizeImage_855911_5.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 928 has memoization hash 5bfa23b8cc6bdbfc81d179573bae1585
parsl.dataflow.memoization INFO: Task 928 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15729440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 928 try 1 launched on executor multi with executor id 836
parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/539fcb88-d75b-4b6e-8708-537ee6010854_characterizeImage_855911_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/539fcb88-d75b-4b6e-8708-537ee6010854_characterizeImage_855911_5.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 793 try 1 failed
parsl.dataflow.dflow ERROR: Task 793 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry ab743d0a9411a9bc83138a49c9fe9623 with result from task 793
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 794 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 794 try 0 failed
parsl.dataflow.dflow INFO: Task 794 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/19659825-6a38-459d-8237-0ee761dd0a3f_calibrate_855911_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/19659825-6a38-459d-8237-0ee761dd0a3f_calibrate_855911_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/3c16f8fe-60c8-44fc-979e-75cc5f8ebe10_characterizeImage_855911_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/3c16f8fe-60c8-44fc-979e-75cc5f8ebe10_characterizeImage_855911_40.stderr
parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eb0b0832480e71dccc8f3c5c2caa1d82 with result from task 289
parsl.dataflow.dflow DEBUG: Task 289 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 290 has memoization hash b455011bf398a6bef8474edf9ba492d5
parsl.dataflow.memoization INFO: Task 290 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 837
parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/43e331d5-226b-4588-b778-ffd157acd4f4_calibrate_855909_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/43e331d5-226b-4588-b778-ffd157acd4f4_calibrate_855909_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/2c6a2059-6b07-40fc-af84-7055d7481011_characterizeImage_855909_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/2c6a2059-6b07-40fc-af84-7055d7481011_characterizeImage_855909_11.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:31 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 61 try 1 failed
parsl.dataflow.dflow ERROR: Task 61 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 671678e604c26484b591d5d13831871e with result from task 61
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 62 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 62 try 0 failed
parsl.dataflow.dflow INFO: Task 62 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/9506370b-d5a5-4c7a-b05c-a36871539be3_calibrate_855910_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/9506370b-d5a5-4c7a-b05c-a36871539be3_calibrate_855910_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/fd018433-8f13-4185-8dd5-1c952533989d_characterizeImage_855910_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/fd018433-8f13-4185-8dd5-1c952533989d_characterizeImage_855910_20.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 137 try 0 failed
parsl.dataflow.dflow INFO: Task 137 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/2a3bf82d-e168-4b5f-a130-e8623b2cfef7_calibrate_855910_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/2a3bf82d-e168-4b5f-a130-e8623b2cfef7_calibrate_855910_4.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 89b171c4fe9324fbb932a23dd3990938
parsl.dataflow.memoization INFO: Task 137 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a41620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 137 try 1 launched on executor multi with executor id 838
parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/2a3bf82d-e168-4b5f-a130-e8623b2cfef7_calibrate_855910_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/2a3bf82d-e168-4b5f-a130-e8623b2cfef7_calibrate_855910_4.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 100 try 1 failed
parsl.dataflow.dflow ERROR: Task 100 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry ca030864dfb5ab35069f670cb4ce2c19 with result from task 100
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 101 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 101 try 0 failed
parsl.dataflow.dflow INFO: Task 101 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/650b2489-6726-4c69-b074-ccdfbe654ea6_calibrate_855910_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/650b2489-6726-4c69-b074-ccdfbe654ea6_calibrate_855910_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/d5e4af58-c2aa-4647-bf1f-61b47cefb824_characterizeImage_855910_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/d5e4af58-c2aa-4647-bf1f-61b47cefb824_characterizeImage_855910_8.stderr
parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a3ef10131e9972f6fafacc9d1ba6b070 with result from task 265
parsl.dataflow.dflow DEBUG: Task 265 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 266 has memoization hash c471f458ad5e84d268039fe506c1f4a7
parsl.dataflow.memoization INFO: Task 266 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 839
parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/223ec596-d277-4fad-bd4f-a63435d42f37_calibrate_855909_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/223ec596-d277-4fad-bd4f-a63435d42f37_calibrate_855909_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/69ea0eae-924f-482c-81db-93e497d5146f_characterizeImage_855909_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/69ea0eae-924f-482c-81db-93e497d5146f_characterizeImage_855909_22.stderr
parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f83a1aaa7a7d9e6e21a2bb943c52e1b1 with result from task 349
parsl.dataflow.dflow DEBUG: Task 349 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 350 has memoization hash e5e2759a0a14e5ad555a76dd67b7ecb6
parsl.dataflow.memoization INFO: Task 350 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16abb4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 840
parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/7626a4f6-e012-44fa-88fa-98a1ac995913_calibrate_855909_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/7626a4f6-e012-44fa-88fa-98a1ac995913_calibrate_855909_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/9527a6ba-e7ee-4870-9eaf-19618fcef374_characterizeImage_855909_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/9527a6ba-e7ee-4870-9eaf-19618fcef374_characterizeImage_855909_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:32 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 463 try 1 failed
parsl.dataflow.dflow ERROR: Task 463 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 5ddfe85142383fc284fb4790b66df0d4 with result from task 463
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 464 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 464 try 0 failed
parsl.dataflow.dflow INFO: Task 464 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/f9851ef9-e301-42d8-a3a6-e5117d2d262e_calibrate_855911_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/f9851ef9-e301-42d8-a3a6-e5117d2d262e_calibrate_855911_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/644a413f-2a07-4ae8-b7c0-8cc140b0e0d8_characterizeImage_855911_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/644a413f-2a07-4ae8-b7c0-8cc140b0e0d8_characterizeImage_855911_18.stderr
parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c6256a479ac4a6dce42410bf92f34d4c with result from task 247
parsl.dataflow.dflow DEBUG: Task 247 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 248 has memoization hash 6d4cbd81c5d64840794f7834c06931c4
parsl.dataflow.memoization INFO: Task 248 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 841
parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/f61911c7-9f60-4ace-aead-fb06c6fd42fb_calibrate_855909_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/f61911c7-9f60-4ace-aead-fb06c6fd42fb_calibrate_855909_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/37cf5ddc-1512-4df8-8352-0b285e7f8602_characterizeImage_855909_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/37cf5ddc-1512-4df8-8352-0b285e7f8602_characterizeImage_855909_29.stderr
parsl.dataflow.dflow INFO: Task 454 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d14a158fbd35f7cc1689a53cbba7940c with result from task 454
parsl.dataflow.dflow DEBUG: Task 454 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 455 has memoization hash 787aabb8e8045467cb69dbe9b1242b31
parsl.dataflow.memoization INFO: Task 455 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 455 try 0 launched on executor multi with executor id 842
parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/69eea3db-0a8f-4f78-b20b-00ed8a0d2519_calibrate_855910_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/69eea3db-0a8f-4f78-b20b-00ed8a0d2519_calibrate_855910_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9aed73d0-a6ae-48dd-a921-fda9dfcd032f_characterizeImage_855910_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9aed73d0-a6ae-48dd-a921-fda9dfcd032f_characterizeImage_855910_23.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 832 try 0 failed
parsl.dataflow.dflow INFO: Task 832 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/d7110b2d-e20c-49a3-b4f7-47908b9769f6_characterizeImage_855910_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/d7110b2d-e20c-49a3-b4f7-47908b9769f6_characterizeImage_855910_14.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 832 has memoization hash 2da1a99999ec608860a7d755b9373590
parsl.dataflow.memoization INFO: Task 832 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15896fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 832 try 1 launched on executor multi with executor id 843
parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/d7110b2d-e20c-49a3-b4f7-47908b9769f6_characterizeImage_855910_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/d7110b2d-e20c-49a3-b4f7-47908b9769f6_characterizeImage_855910_14.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:32 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 310 try 1 failed
parsl.dataflow.dflow ERROR: Task 310 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 116741d08cc736010feb115f1d32f12f with result from task 310
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 311 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 311 try 0 failed
parsl.dataflow.dflow INFO: Task 311 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/29075ad2-0bd1-49ff-a436-6a48303599f0_calibrate_855911_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/29075ad2-0bd1-49ff-a436-6a48303599f0_calibrate_855911_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/19e593de-557a-4ee4-b389-d28b2d53cc5e_characterizeImage_855911_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/19e593de-557a-4ee4-b389-d28b2d53cc5e_characterizeImage_855911_21.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 802 try 1 failed
parsl.dataflow.dflow ERROR: Task 802 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 31c89347931d428ba7178e70dbd3a85a with result from task 802
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 803 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 803 try 0 failed
parsl.dataflow.dflow INFO: Task 803 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/e8e939ee-4332-4696-9340-a86425cbcac2_calibrate_855910_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/e8e939ee-4332-4696-9340-a86425cbcac2_calibrate_855910_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a3f165c3-e409-4667-a596-123d9ae5bbee_characterizeImage_855910_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a3f165c3-e409-4667-a596-123d9ae5bbee_characterizeImage_855910_39.stderr
parsl.dataflow.dflow INFO: Task 982 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0aba1bb3007ec4bd33502f5057eb9307 with result from task 982
parsl.dataflow.dflow DEBUG: Task 982 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 983 has memoization hash 72e458a4b2c2b3230805bc2c5b56b32b
parsl.dataflow.memoization INFO: Task 983 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 983 try 0 launched on executor multi with executor id 844
parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/c6967517-8e3b-44c1-87d3-e744febb606c_calibrate_855911_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/c6967517-8e3b-44c1-87d3-e744febb606c_calibrate_855911_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/bac4de84-eed6-4e4b-931b-1c29f3f689c8_characterizeImage_855911_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/bac4de84-eed6-4e4b-931b-1c29f3f689c8_characterizeImage_855911_30.stderr
parsl.dataflow.dflow INFO: Task 1057 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 66c2f1977015bd213c1bd8023c993de9 with result from task 1057
parsl.dataflow.dflow DEBUG: Task 1057 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1058 has memoization hash 3b6af49496c4d80760fa289ee6245d99
parsl.dataflow.memoization INFO: Task 1058 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1058 try 0 launched on executor multi with executor id 845
parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/9951a541-ed1c-4dd4-8424-4bb33b7a3c6c_calibrate_855910_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/9951a541-ed1c-4dd4-8424-4bb33b7a3c6c_calibrate_855910_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/d3301da5-a335-4071-a778-ad8a68dba0c8_characterizeImage_855910_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/d3301da5-a335-4071-a778-ad8a68dba0c8_characterizeImage_855910_48.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:33 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 1105 try 1 failed
parsl.dataflow.dflow ERROR: Task 1105 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry f4e736cebf1978a7b05b4b85446d4a4c with result from task 1105
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 1106 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 1106 try 0 failed
parsl.dataflow.dflow INFO: Task 1106 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/e05675fe-42c8-4e1f-883e-28b9c96a587d_calibrate_855910_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/e05675fe-42c8-4e1f-883e-28b9c96a587d_calibrate_855910_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/34a4e514-e396-44a4-8579-280c35a0e37e_characterizeImage_855910_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/34a4e514-e396-44a4-8579-280c35a0e37e_characterizeImage_855910_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:34 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 325 try 1 failed
parsl.dataflow.dflow ERROR: Task 325 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 07f04ed70709f8032682cf9daa92e3db with result from task 325
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 326 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 326 try 0 failed
parsl.dataflow.dflow INFO: Task 326 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/dbdfe4d7-dd28-4bbb-8560-928a01c0f8c8_calibrate_855911_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/dbdfe4d7-dd28-4bbb-8560-928a01c0f8c8_calibrate_855911_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c665b36c-f2f1-4331-99fc-e27f91d5f12e_characterizeImage_855911_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c665b36c-f2f1-4331-99fc-e27f91d5f12e_characterizeImage_855911_61.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 985 try 0 failed
parsl.dataflow.dflow INFO: Task 985 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c92cc6f5-cf4f-4d64-9779-b90e3f42d0fc_characterizeImage_855911_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c92cc6f5-cf4f-4d64-9779-b90e3f42d0fc_characterizeImage_855911_58.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 985 has memoization hash 07e9e47bcfc831eff8f4d5e013c43d01
parsl.dataflow.memoization INFO: Task 985 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 985 try 1 launched on executor multi with executor id 846
parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c92cc6f5-cf4f-4d64-9779-b90e3f42d0fc_characterizeImage_855911_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c92cc6f5-cf4f-4d64-9779-b90e3f42d0fc_characterizeImage_855911_58.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 532 try 1 failed
parsl.dataflow.dflow ERROR: Task 532 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 7ae61edb24360387beb85e59a58531cc with result from task 532
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 533 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 533 try 0 failed
parsl.dataflow.dflow INFO: Task 533 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/3e84277e-0c0c-4558-9fa6-4de1f6286e1b_calibrate_855910_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/3e84277e-0c0c-4558-9fa6-4de1f6286e1b_calibrate_855910_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b28ff9c2-f409-4140-8f81-68513963b66a_characterizeImage_855910_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b28ff9c2-f409-4140-8f81-68513963b66a_characterizeImage_855910_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:34 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 54924bd8e85d772764e81ae558f96a49 with result from task 319
parsl.dataflow.dflow DEBUG: Task 319 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 320 has memoization hash df9178c0deb6bf91cc200e03d110ca01
parsl.dataflow.memoization INFO: Task 320 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 847
parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/7f84eb19-d9b2-471b-a039-839c1b925310_calibrate_855909_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/7f84eb19-d9b2-471b-a039-839c1b925310_calibrate_855909_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/5b315c1d-f32c-4cde-830c-d6a7cb816ed6_characterizeImage_855909_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/5b315c1d-f32c-4cde-830c-d6a7cb816ed6_characterizeImage_855909_36.stderr
parsl.dataflow.dflow INFO: Task 493 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5e75331aa359a1f8ab04cc0a973468c1 with result from task 493
parsl.dataflow.dflow DEBUG: Task 493 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 494 has memoization hash fe4e3fd9cf69e4d1c7e90747d271cefb
parsl.dataflow.memoization INFO: Task 494 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15944b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 494 try 0 launched on executor multi with executor id 848
parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/15f7ee43-e8e8-49ed-933d-fbf87de3ec7e_calibrate_855911_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/15f7ee43-e8e8-49ed-933d-fbf87de3ec7e_calibrate_855911_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/b09b9d40-9292-466c-9491-6721b6df807d_characterizeImage_855911_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/b09b9d40-9292-466c-9491-6721b6df807d_characterizeImage_855911_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:35 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d33ef21e16fc9ade18cc01e249ed93c5 with result from task 277
parsl.dataflow.dflow DEBUG: Task 277 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 77418da00e107e98d91f7e21e160d790
parsl.dataflow.memoization INFO: Task 278 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 849
parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/f1f01e01-2819-48d4-97be-cb18f8a0396d_calibrate_855911_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/f1f01e01-2819-48d4-97be-cb18f8a0396d_calibrate_855911_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/7fcf69f2-8a37-47ff-bd9d-99e0d12220bb_characterizeImage_855911_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/7fcf69f2-8a37-47ff-bd9d-99e0d12220bb_characterizeImage_855911_34.stderr
parsl.dataflow.dflow INFO: Task 979 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cec063361741c867881f5701030218c1 with result from task 979
parsl.dataflow.dflow DEBUG: Task 979 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 980 has memoization hash 346228143fc9871a9d1b80cad97ca28d
parsl.dataflow.memoization INFO: Task 980 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 980 try 0 launched on executor multi with executor id 850
parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/139674f8-487f-4e75-b947-069d5b995691_calibrate_855911_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/139674f8-487f-4e75-b947-069d5b995691_calibrate_855911_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/33fa17d6-d782-4649-9e77-241c861bcbae_characterizeImage_855911_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/33fa17d6-d782-4649-9e77-241c861bcbae_characterizeImage_855911_27.stderr
parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d5cf1e2b92bd1f8c58cc6f70de9283b5 with result from task 283
parsl.dataflow.dflow DEBUG: Task 283 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 284 has memoization hash 5e45d4882a04a974c1a6b45d6fe6571c
parsl.dataflow.memoization INFO: Task 284 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 851
parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/290dc026-69b5-4843-b2e5-5d2c98504474_calibrate_855910_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/290dc026-69b5-4843-b2e5-5d2c98504474_calibrate_855910_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/f88194eb-8cbd-4bcd-877a-edaec2c0ce55_characterizeImage_855910_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/f88194eb-8cbd-4bcd-877a-edaec2c0ce55_characterizeImage_855910_41.stderr
parsl.dataflow.dflow INFO: Task 898 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d2254b6fbc4f03b3296d8ef5af7b17a2 with result from task 898
parsl.dataflow.dflow DEBUG: Task 898 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 899 has memoization hash 9d05fbcb10181f8470d48c0c3c1ac35d
parsl.dataflow.memoization INFO: Task 899 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156eb2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 899 try 0 launched on executor multi with executor id 852
parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6c3131ac-dbeb-4634-8000-50c33395ac1d_calibrate_855910_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6c3131ac-dbeb-4634-8000-50c33395ac1d_calibrate_855910_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/ea1ab206-74a7-46ee-bd6b-e656cbde3137_characterizeImage_855910_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/ea1ab206-74a7-46ee-bd6b-e656cbde3137_characterizeImage_855910_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:35 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 538 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fa226f7c646149c0bf71633ea6dada25 with result from task 538
parsl.dataflow.dflow DEBUG: Task 538 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 539 has memoization hash a6f35acad06134392477b1c2b65f73a5
parsl.dataflow.memoization INFO: Task 539 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15947a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 539 try 0 launched on executor multi with executor id 853
parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/72096477-5c4e-4358-8ece-0747cb3f291f_calibrate_855911_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/72096477-5c4e-4358-8ece-0747cb3f291f_calibrate_855911_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/4a1425ab-70e7-466e-95cd-29a7c8aae0b1_characterizeImage_855911_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/4a1425ab-70e7-466e-95cd-29a7c8aae0b1_characterizeImage_855911_51.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:36 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:37 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 670 try 1 failed
parsl.dataflow.dflow ERROR: Task 670 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 35dde2a4d9ccd4f23e5e0dfc62840a90 with result from task 670
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 671 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 671 try 0 failed
parsl.dataflow.dflow INFO: Task 671 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/36520d50-887c-459d-bfa5-1dc9dc4c2f4a_calibrate_855910_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/36520d50-887c-459d-bfa5-1dc9dc4c2f4a_calibrate_855910_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/e25fe73f-595a-4f0f-8204-3dc027466c20_characterizeImage_855910_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/e25fe73f-595a-4f0f-8204-3dc027466c20_characterizeImage_855910_19.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 664 try 1 failed
parsl.dataflow.dflow ERROR: Task 664 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 983270ae2cf3bab3fabbfa493f7cfb76 with result from task 664
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 665 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 665 try 0 failed
parsl.dataflow.dflow INFO: Task 665 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/d1c59bc5-33e1-40de-adcf-fb2d7969fb88_calibrate_855910_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/d1c59bc5-33e1-40de-adcf-fb2d7969fb88_calibrate_855910_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/630877c9-a022-4ed5-97ac-e679b28da27f_characterizeImage_855910_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/630877c9-a022-4ed5-97ac-e679b28da27f_characterizeImage_855910_46.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 589 try 1 failed
parsl.dataflow.dflow ERROR: Task 589 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 98eade9ac5578605139a9f6ded2dba22 with result from task 589
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 590 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 590 try 0 failed
parsl.dataflow.dflow INFO: Task 590 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/da00e7a2-3aec-41fe-8b43-945876eeac7e_calibrate_855911_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/da00e7a2-3aec-41fe-8b43-945876eeac7e_calibrate_855911_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/dbcece99-9605-4365-9c86-3edcd40507ab_characterizeImage_855911_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/dbcece99-9605-4365-9c86-3edcd40507ab_characterizeImage_855911_3.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:37 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1aaedeabe0145fe666808f4d0d5a0ce5 with result from task 322
parsl.dataflow.dflow DEBUG: Task 322 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 8b30ee715ffa0e6b3a65d01a8c0532db
parsl.dataflow.memoization INFO: Task 323 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 854
parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/0eee26b4-1b3b-4c12-9481-8c88c1fb2691_calibrate_855909_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/0eee26b4-1b3b-4c12-9481-8c88c1fb2691_calibrate_855909_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/c992bb54-85b1-4bcd-bdc5-5ed286804fdf_characterizeImage_855909_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/c992bb54-85b1-4bcd-bdc5-5ed286804fdf_characterizeImage_855909_17.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 55 try 1 failed
parsl.dataflow.dflow ERROR: Task 55 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 2ae285dffcef1e0e82b3bb35f8a3a3cf with result from task 55
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 56 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 56 try 0 failed
parsl.dataflow.dflow INFO: Task 56 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/5a26c2e0-c91d-477e-b3d4-f64ab89935a3_calibrate_855911_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/5a26c2e0-c91d-477e-b3d4-f64ab89935a3_calibrate_855911_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/197d3d62-39c7-4303-b5c4-6f03c71107cb_characterizeImage_855911_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/197d3d62-39c7-4303-b5c4-6f03c71107cb_characterizeImage_855911_22.stderr
parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7466995a82c743d617d8aac6eb0df392 with result from task 127
parsl.dataflow.dflow DEBUG: Task 127 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 128 has memoization hash a9e21cea46d38bf050616593983eef4b
parsl.dataflow.memoization INFO: Task 128 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a40cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 855
parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/c491572c-097b-4878-9d6a-49b8ad3ed565_calibrate_855909_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/c491572c-097b-4878-9d6a-49b8ad3ed565_calibrate_855909_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/7d0588e2-8f78-4848-8899-7486d9b8b2b3_characterizeImage_855909_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/7d0588e2-8f78-4848-8899-7486d9b8b2b3_characterizeImage_855909_60.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 1004 try 0 failed
parsl.dataflow.dflow INFO: Task 1004 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/c0b60462-6b9e-4161-9fe1-300e9185d26d_calibrate_855911_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/c0b60462-6b9e-4161-9fe1-300e9185d26d_calibrate_855911_32.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1004 has memoization hash adf6cb0f74bf726d49569545af5a744c
parsl.dataflow.memoization INFO: Task 1004 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1004 try 1 launched on executor multi with executor id 856
parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/c0b60462-6b9e-4161-9fe1-300e9185d26d_calibrate_855911_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/c0b60462-6b9e-4161-9fe1-300e9185d26d_calibrate_855911_32.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:38 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 571 try 1 failed
parsl.dataflow.dflow ERROR: Task 571 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry cdac144bbb8353fc5cabbca7a2b0df76 with result from task 571
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 572 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 572 try 0 failed
parsl.dataflow.dflow INFO: Task 572 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/72296958-18e8-4cb0-8e3f-3fb2564cb3e5_calibrate_855911_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/72296958-18e8-4cb0-8e3f-3fb2564cb3e5_calibrate_855911_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/d2050731-17bb-472a-a78e-a1f0abadb25f_characterizeImage_855911_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/d2050731-17bb-472a-a78e-a1f0abadb25f_characterizeImage_855911_39.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:38 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 724 try 1 failed
parsl.dataflow.dflow ERROR: Task 724 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 539a1468b63de63caaefd742a0154733 with result from task 724
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 725 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 725 try 0 failed
parsl.dataflow.dflow INFO: Task 725 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/fe5084de-e28e-4949-9c29-7b64b9ebf990_calibrate_855911_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/fe5084de-e28e-4949-9c29-7b64b9ebf990_calibrate_855911_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/86f749f5-3fbf-4fb8-be29-20a0377b7713_characterizeImage_855911_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/86f749f5-3fbf-4fb8-be29-20a0377b7713_characterizeImage_855911_6.stderr
parsl.dataflow.dflow INFO: Task 760 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1a97784f91973dec931145df95f01d7f with result from task 760
parsl.dataflow.dflow DEBUG: Task 760 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 761 has memoization hash 69e64f2d7c181fc6cc02920d5cfaa61a
parsl.dataflow.memoization INFO: Task 761 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 761 try 0 launched on executor multi with executor id 857
parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/b385a0f6-322f-47c8-b109-0f442c42fa5e_calibrate_855910_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/b385a0f6-322f-47c8-b109-0f442c42fa5e_calibrate_855910_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7194435b-70cf-4770-9caa-7e050318a1d4_characterizeImage_855910_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7194435b-70cf-4770-9caa-7e050318a1d4_characterizeImage_855910_30.stderr
parsl.dataflow.dflow INFO: Task 436 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f6f93299db031d64f245bb505710a6ba with result from task 436
parsl.dataflow.dflow DEBUG: Task 436 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 437 has memoization hash c267caf8d08b8b601359b90b2f913e6d
parsl.dataflow.memoization INFO: Task 437 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 437 try 0 launched on executor multi with executor id 858
parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/b5cf2999-9fc6-440a-87e9-830137771018_calibrate_855911_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/b5cf2999-9fc6-440a-87e9-830137771018_calibrate_855911_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/80762dfd-6863-4cbe-8166-6ec8cdc2870a_characterizeImage_855911_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/80762dfd-6863-4cbe-8166-6ec8cdc2870a_characterizeImage_855911_60.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 640 try 1 failed
parsl.dataflow.dflow ERROR: Task 640 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry b82150ed2b6c8575218c021fafb8c554 with result from task 640
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 641 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 641 try 0 failed
parsl.dataflow.dflow INFO: Task 641 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/39c6d3a3-7eed-4d84-b164-b1be0d4eee06_calibrate_855910_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/39c6d3a3-7eed-4d84-b164-b1be0d4eee06_calibrate_855910_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/87fffcf9-43c1-4b40-8db9-3ecba65a35b5_characterizeImage_855910_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/87fffcf9-43c1-4b40-8db9-3ecba65a35b5_characterizeImage_855910_25.stderr
parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 57547085f1eeb75b96682f005ae8d80a with result from task 304
parsl.dataflow.dflow DEBUG: Task 304 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 305 has memoization hash 1c9cc27681a79947383e9036fb625d31
parsl.dataflow.memoization INFO: Task 305 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 859
parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/21e1bdd4-1597-4c3e-8ce9-a231a75a2da1_calibrate_855909_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/21e1bdd4-1597-4c3e-8ce9-a231a75a2da1_calibrate_855909_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/80c1e7a7-676b-4941-9117-406eb14c20cf_characterizeImage_855909_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/80c1e7a7-676b-4941-9117-406eb14c20cf_characterizeImage_855909_5.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 694 try 1 failed
parsl.dataflow.dflow ERROR: Task 694 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry e028cd437b0013743831baa511937486 with result from task 694
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 695 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 695 try 0 failed
parsl.dataflow.dflow INFO: Task 695 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/e73ad9b5-7ed4-4b9a-9893-3f0b8af13f41_calibrate_855911_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/e73ad9b5-7ed4-4b9a-9893-3f0b8af13f41_calibrate_855911_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/99231e1d-b415-4de5-8510-c098fb36585f_characterizeImage_855911_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/99231e1d-b415-4de5-8510-c098fb36585f_characterizeImage_855911_44.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 616 try 1 failed
parsl.dataflow.dflow ERROR: Task 616 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry e96d371890aa491733302d21a1b67712 with result from task 616
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 617 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 617 try 0 failed
parsl.dataflow.dflow INFO: Task 617 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/d6b985a8-f163-4032-8cef-c08aa848d112_calibrate_855911_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/d6b985a8-f163-4032-8cef-c08aa848d112_calibrate_855911_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/314d9a2e-afb2-44b7-aae0-d60cc3469b3a_characterizeImage_855911_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/314d9a2e-afb2-44b7-aae0-d60cc3469b3a_characterizeImage_855911_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:39 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6f1fe045f887b8a4f9535a310134a3b3 with result from task 49
parsl.dataflow.dflow DEBUG: Task 49 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 7fb7a8136d050578eea08b0a84c4111f
parsl.dataflow.memoization INFO: Task 50 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169ffb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 860
parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/9c673469-c840-4ea4-955a-2641cff95a23_calibrate_855909_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/9c673469-c840-4ea4-955a-2641cff95a23_calibrate_855909_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/fb9b9416-82e6-4ec8-aa76-4226497178bc_characterizeImage_855909_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/fb9b9416-82e6-4ec8-aa76-4226497178bc_characterizeImage_855909_39.stderr
parsl.dataflow.dflow INFO: Task 757 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f31c90d08c8b7a684d18e925f1084b88 with result from task 757
parsl.dataflow.dflow DEBUG: Task 757 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 758 has memoization hash 9e2c0686fae8f327dc644dc3b9a880bf
parsl.dataflow.memoization INFO: Task 758 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15859f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 758 try 0 launched on executor multi with executor id 861
parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/c59ed363-f5be-443f-86d8-4cad85bbdf47_calibrate_855910_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/c59ed363-f5be-443f-86d8-4cad85bbdf47_calibrate_855910_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/1b0fa141-3af7-4616-937f-9bafc2349196_characterizeImage_855910_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/1b0fa141-3af7-4616-937f-9bafc2349196_characterizeImage_855910_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:39 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 293 try 0 failed
parsl.dataflow.dflow INFO: Task 293 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/1e37b574-52b0-418c-a9db-600752e5554b_calibrate_855911_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/1e37b574-52b0-418c-a9db-600752e5554b_calibrate_855911_14.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 293 has memoization hash c1340757feaff4f75202df85c1400c08
parsl.dataflow.memoization INFO: Task 293 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 293 try 1 launched on executor multi with executor id 862
parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/1e37b574-52b0-418c-a9db-600752e5554b_calibrate_855911_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/1e37b574-52b0-418c-a9db-600752e5554b_calibrate_855911_14.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 709 try 1 failed
parsl.dataflow.dflow ERROR: Task 709 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 37005d6a6d1cd9fb6effc35f1fc4054d with result from task 709
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 710 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 710 try 0 failed
parsl.dataflow.dflow INFO: Task 710 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/5f9cb9e3-24e0-402c-8328-0677739308bd_calibrate_855910_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/5f9cb9e3-24e0-402c-8328-0677739308bd_calibrate_855910_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/41f4abe5-116f-4461-bb35-e4782aed1809_characterizeImage_855910_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/41f4abe5-116f-4461-bb35-e4782aed1809_characterizeImage_855910_40.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 1042 try 1 failed
parsl.dataflow.dflow ERROR: Task 1042 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry c383d405c4b25e05360df755ee7af784 with result from task 1042
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 1043 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 1043 try 0 failed
parsl.dataflow.dflow INFO: Task 1043 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/bcca9f5b-1177-4485-ab3c-725fdb552536_calibrate_855911_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/bcca9f5b-1177-4485-ab3c-725fdb552536_calibrate_855911_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/7570243c-6ca2-44db-8516-bd1db9b982f4_characterizeImage_855911_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/7570243c-6ca2-44db-8516-bd1db9b982f4_characterizeImage_855911_45.stderr
parsl.dataflow.dflow INFO: Task 526 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c5b2d01d66b6d5b7e2d59414a9aed428 with result from task 526
parsl.dataflow.dflow DEBUG: Task 526 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 527 has memoization hash dba7e236a38194c720b49c96aa30b216
parsl.dataflow.memoization INFO: Task 527 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15946de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 527 try 0 launched on executor multi with executor id 863
parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/6aab1a49-b9a9-4326-bb5e-515cc7609091_calibrate_855911_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/6aab1a49-b9a9-4326-bb5e-515cc7609091_calibrate_855911_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/d365bfd2-cfd5-44fd-b3bf-2795d8640625_characterizeImage_855911_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/d365bfd2-cfd5-44fd-b3bf-2795d8640625_characterizeImage_855911_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:40 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 712 try 1 failed
parsl.dataflow.dflow ERROR: Task 712 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry d953d888a7e2164ad5898ef37639131d with result from task 712
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 713 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 713 try 0 failed
parsl.dataflow.dflow INFO: Task 713 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/5dd11cfa-d4fc-4840-ad88-fb56863fc523_calibrate_855910_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/5dd11cfa-d4fc-4840-ad88-fb56863fc523_calibrate_855910_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b8d5e5fe-2daf-48fe-949a-bea601c3e3df_characterizeImage_855910_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b8d5e5fe-2daf-48fe-949a-bea601c3e3df_characterizeImage_855910_13.stderr
parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8d5e876f730b3a02c3f653f54e1d311a with result from task 14
parsl.dataflow.dflow DEBUG: Task 14 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/2fce3322-7467-48eb-902b-7ce876a1e6ba_calibrate_855911_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/2fce3322-7467-48eb-902b-7ce876a1e6ba_calibrate_855911_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:41 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 751 try 1 failed
parsl.dataflow.dflow ERROR: Task 751 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 3014026d02ff6831ef5034427d216632 with result from task 751
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 752 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 752 try 0 failed
parsl.dataflow.dflow INFO: Task 752 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/6593eb5a-0019-4331-8e56-6d5b74c42bac_calibrate_855911_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/6593eb5a-0019-4331-8e56-6d5b74c42bac_calibrate_855911_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/da0d1552-f18a-4d65-b727-c59d69ca334a_characterizeImage_855911_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/da0d1552-f18a-4d65-b727-c59d69ca334a_characterizeImage_855911_46.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 520 try 1 failed
parsl.dataflow.dflow ERROR: Task 520 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry af4f7ecce07b87824f97859da083cecb with result from task 520
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 521 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 521 try 0 failed
parsl.dataflow.dflow INFO: Task 521 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/d8f798af-1bdd-4ecd-9335-881c7a6b9bc3_calibrate_855911_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/d8f798af-1bdd-4ecd-9335-881c7a6b9bc3_calibrate_855911_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/01c702af-c157-42c4-9e9d-62292b95238f_characterizeImage_855911_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/01c702af-c157-42c4-9e9d-62292b95238f_characterizeImage_855911_11.stderr
parsl.dataflow.dflow INFO: Task 418 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1cc81a05f69f9c1952a4bc7fbc0a7bd8 with result from task 418
parsl.dataflow.dflow DEBUG: Task 418 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 419 has memoization hash b44cdd0282b06ea0866a86becddaa0a1
parsl.dataflow.memoization INFO: Task 419 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 419 try 0 launched on executor multi with executor id 864
parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/a4aad4bb-ea20-43b8-b79d-3fcce866a3db_calibrate_855909_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/a4aad4bb-ea20-43b8-b79d-3fcce866a3db_calibrate_855909_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/3c3d465f-c012-4488-a809-61169c68a510_characterizeImage_855909_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/3c3d465f-c012-4488-a809-61169c68a510_characterizeImage_855909_41.stderr
parsl.dataflow.dflow INFO: Task 697 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bb0bc9d7594dc47381745ce0f6c10760 with result from task 697
parsl.dataflow.dflow DEBUG: Task 697 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 698 has memoization hash 9271daf0880bb243ae7a79378bed0c57
parsl.dataflow.memoization INFO: Task 698 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 698 try 0 launched on executor multi with executor id 865
parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/60ccd845-d161-4903-943c-3795d143d14f_calibrate_855910_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/60ccd845-d161-4903-943c-3795d143d14f_calibrate_855910_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/66ebc2c9-521d-4d35-a305-f74b9a19e580_characterizeImage_855910_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/66ebc2c9-521d-4d35-a305-f74b9a19e580_characterizeImage_855910_21.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 146 try 0 failed
parsl.dataflow.dflow INFO: Task 146 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/d0b16eaa-7653-4e4e-8d2d-6b5e09af82b8_calibrate_855910_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/d0b16eaa-7653-4e4e-8d2d-6b5e09af82b8_calibrate_855910_34.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 676b73370afe2f1ef4f39f09ee6e1c1b
parsl.dataflow.memoization INFO: Task 146 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a41f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 146 try 1 launched on executor multi with executor id 866
parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/d0b16eaa-7653-4e4e-8d2d-6b5e09af82b8_calibrate_855910_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/d0b16eaa-7653-4e4e-8d2d-6b5e09af82b8_calibrate_855910_34.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:41 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 913 try 1 failed
parsl.dataflow.dflow ERROR: Task 913 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 0019dd4534ec95365f8fdd85ca1abd1b with result from task 913
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 914 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 914 try 0 failed
parsl.dataflow.dflow INFO: Task 914 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/b28c0698-8b82-4487-8711-5cb29b89722e_calibrate_855910_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/b28c0698-8b82-4487-8711-5cb29b89722e_calibrate_855910_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/cb8d9282-f6ee-47b2-b580-a53006f2c1ce_characterizeImage_855910_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/cb8d9282-f6ee-47b2-b580-a53006f2c1ce_characterizeImage_855910_51.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 721 try 1 failed
parsl.dataflow.dflow ERROR: Task 721 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 900a9cada33524e4e1d7252a074c1b34 with result from task 721
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 722 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 722 try 0 failed
parsl.dataflow.dflow INFO: Task 722 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/b6c86627-8956-44d9-b5f4-eea8c2227342_calibrate_855911_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/b6c86627-8956-44d9-b5f4-eea8c2227342_calibrate_855911_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/cdb1cee3-7fa1-4ebc-bcf0-a9ef1ae1cfec_characterizeImage_855911_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/cdb1cee3-7fa1-4ebc-bcf0-a9ef1ae1cfec_characterizeImage_855911_13.stderr
parsl.dataflow.dflow INFO: Task 742 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cad572f80e7451643ccc9b10f6d4f330 with result from task 742
parsl.dataflow.dflow DEBUG: Task 742 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 743 has memoization hash 7e286da1999a4d684b02d375db7f15bc
parsl.dataflow.memoization INFO: Task 743 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15858fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 743 try 0 launched on executor multi with executor id 867
parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6aafe7b7-5807-48e7-a052-46ad4af88d13_calibrate_855910_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6aafe7b7-5807-48e7-a052-46ad4af88d13_calibrate_855910_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/ee15b11f-7e12-460c-b283-16aa1a19a90b_characterizeImage_855910_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/ee15b11f-7e12-460c-b283-16aa1a19a90b_characterizeImage_855910_3.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:42 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 407 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 17e6d9d94e8a714dfe6501687656f916 with result from task 407
parsl.dataflow.dflow DEBUG: Task 407 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/91250870-cd13-409c-9e8d-751891514cdd_calibrate_855911_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/91250870-cd13-409c-9e8d-751891514cdd_calibrate_855911_10.stderr
parsl.dataflow.dflow INFO: Task 442 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 44cd07ae95eee81d77182c83d3f9251c with result from task 442
parsl.dataflow.dflow DEBUG: Task 442 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 443 has memoization hash 7038937d9c07bbb992cd3b0036148455
parsl.dataflow.memoization INFO: Task 443 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 443 try 0 launched on executor multi with executor id 868
parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6cf5d066-ac35-414d-84f4-66d62329c8a4_calibrate_855910_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6cf5d066-ac35-414d-84f4-66d62329c8a4_calibrate_855910_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/bb1c1232-a3c9-4514-8fc9-3cb40eb30104_characterizeImage_855910_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/bb1c1232-a3c9-4514-8fc9-3cb40eb30104_characterizeImage_855910_60.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 1001 try 0 failed
parsl.dataflow.dflow INFO: Task 1001 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/851e6f2e-f498-4375-ba09-41b714b0c36b_calibrate_855911_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/851e6f2e-f498-4375-ba09-41b714b0c36b_calibrate_855911_35.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1001 has memoization hash 7cabc7b4f1da8c753a2e3846d3a95ba2
parsl.dataflow.memoization INFO: Task 1001 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1001 try 1 launched on executor multi with executor id 869
parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/851e6f2e-f498-4375-ba09-41b714b0c36b_calibrate_855911_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/851e6f2e-f498-4375-ba09-41b714b0c36b_calibrate_855911_35.stderr
parsl.dataflow.dflow INFO: Task 1108 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry beedc52c983e98c044d80cbe14323efd with result from task 1108
parsl.dataflow.dflow DEBUG: Task 1108 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1109 has memoization hash 9cfc81dfab522799a2ca86b80fbc6050
parsl.dataflow.memoization INFO: Task 1109 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1560cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1109 try 0 launched on executor multi with executor id 870
parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/8bfb21c8-cdf9-4f9b-9cb9-873ef10d015c_calibrate_855911_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/8bfb21c8-cdf9-4f9b-9cb9-873ef10d015c_calibrate_855911_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/cf847fe0-7e23-4dbc-883d-794cea1a8aec_characterizeImage_855911_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/cf847fe0-7e23-4dbc-883d-794cea1a8aec_characterizeImage_855911_56.stderr
parsl.dataflow.dflow INFO: Task 781 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 587174573fc0877ed7a61c0e8a10e16a with result from task 781
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:42 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow DEBUG: Task 781 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 782 has memoization hash 82d9183aab6665929154bcb9ce61b1e4
parsl.dataflow.memoization INFO: Task 782 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 782 try 0 launched on executor multi with executor id 871
parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/ffa5ab2c-891f-4fc1-a396-f62629a6703f_calibrate_855911_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/ffa5ab2c-891f-4fc1-a396-f62629a6703f_calibrate_855911_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/f63b146a-ab5d-4c8e-b646-bf1e3a8ab01e_characterizeImage_855911_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/f63b146a-ab5d-4c8e-b646-bf1e3a8ab01e_characterizeImage_855911_24.stderr
parsl.dataflow.dflow INFO: Task 680 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 835dbf2b34cefd178e6349701b0aac0f with result from task 680
parsl.dataflow.dflow DEBUG: Task 680 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/014bfeba-e23c-440e-a439-302ceb85b836_calibrate_855911_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/014bfeba-e23c-440e-a439-302ceb85b836_calibrate_855911_54.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 928 try 1 failed
parsl.dataflow.dflow ERROR: Task 928 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 5bfa23b8cc6bdbfc81d179573bae1585 with result from task 928
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 929 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 929 try 0 failed
parsl.dataflow.dflow INFO: Task 929 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/7cad5fb8-b336-42ee-b358-fd30f45a19e3_calibrate_855911_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/7cad5fb8-b336-42ee-b358-fd30f45a19e3_calibrate_855911_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/539fcb88-d75b-4b6e-8708-537ee6010854_characterizeImage_855911_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/539fcb88-d75b-4b6e-8708-537ee6010854_characterizeImage_855911_5.stderr
parsl.dataflow.dflow INFO: Task 523 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 14da39d7ae7ca6ff37e8aa91e63e56fc with result from task 523
parsl.dataflow.dflow DEBUG: Task 523 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 524 has memoization hash 417283f4039f481ae2d3d3d586b7cfc9
parsl.dataflow.memoization INFO: Task 524 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15946ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 524 try 0 launched on executor multi with executor id 872
parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/d664740a-86e0-4343-8188-4278e80bf8ad_calibrate_855909_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/d664740a-86e0-4343-8188-4278e80bf8ad_calibrate_855909_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/6704df77-76f2-4617-a4e9-a2c039cec75f_characterizeImage_855909_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/6704df77-76f2-4617-a4e9-a2c039cec75f_characterizeImage_855909_14.stderr
parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f53c12806536d1a47d8fcee38131e21b with result from task 98
parsl.dataflow.dflow DEBUG: Task 98 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/7f728b61-7bd1-447a-a446-2b59f819f43f_calibrate_855911_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/7f728b61-7bd1-447a-a446-2b59f819f43f_calibrate_855911_55.stderr
parsl.dataflow.dflow INFO: Task 916 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 78e53eac52ef3737f534ad136489cd4e with result from task 916
parsl.dataflow.dflow DEBUG: Task 916 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 917 has memoization hash 2032390fe3805092b9d95881a0359710
parsl.dataflow.memoization INFO: Task 917 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157285e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 917 try 0 launched on executor multi with executor id 873
parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/ca403cb0-5807-46b9-b676-df869a553e85_calibrate_855910_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/ca403cb0-5807-46b9-b676-df869a553e85_calibrate_855910_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/0bdf3ccb-bd3f-403f-bbec-3370ee7bb05f_characterizeImage_855910_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/0bdf3ccb-bd3f-403f-bbec-3370ee7bb05f_characterizeImage_855910_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:43 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 730 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 93a774ead62126e57411be69fd0b5948 with result from task 730
parsl.dataflow.dflow DEBUG: Task 730 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 731 has memoization hash 3901c9562a0568e9cc2ef3c73d5d083e
parsl.dataflow.memoization INFO: Task 731 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15858360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 731 try 0 launched on executor multi with executor id 874
parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/c5fd6e4d-3831-46c8-a994-9b95d770ab26_calibrate_855911_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/c5fd6e4d-3831-46c8-a994-9b95d770ab26_calibrate_855911_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/1717e1ed-b56e-4a49-a425-b54344460d32_characterizeImage_855911_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/1717e1ed-b56e-4a49-a425-b54344460d32_characterizeImage_855911_26.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 832 try 1 failed
parsl.dataflow.dflow ERROR: Task 832 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 2da1a99999ec608860a7d755b9373590 with result from task 832
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 833 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 833 try 0 failed
parsl.dataflow.dflow INFO: Task 833 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/e914f04c-6eb3-4c6e-9948-c6b3999f3aec_calibrate_855910_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/e914f04c-6eb3-4c6e-9948-c6b3999f3aec_calibrate_855910_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/d7110b2d-e20c-49a3-b4f7-47908b9769f6_characterizeImage_855910_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/d7110b2d-e20c-49a3-b4f7-47908b9769f6_characterizeImage_855910_14.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:44 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 541 try 1 failed
parsl.dataflow.dflow ERROR: Task 541 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 2afeeab34b1fa3be3465709379808987 with result from task 541
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 542 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 542 try 0 failed
parsl.dataflow.dflow INFO: Task 542 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/f3163eb1-0a08-4618-b0de-31495af829b1_calibrate_855911_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/f3163eb1-0a08-4618-b0de-31495af829b1_calibrate_855911_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/2f717859-0942-4d79-b214-59d39385fa48_characterizeImage_855911_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/2f717859-0942-4d79-b214-59d39385fa48_characterizeImage_855911_53.stderr
parsl.dataflow.dflow INFO: Task 808 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e3067e41090f5a05a07d7b54ea10adbe with result from task 808
parsl.dataflow.dflow DEBUG: Task 808 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 809 has memoization hash a18109fc2eed485952c67e95892e3673
parsl.dataflow.memoization INFO: Task 809 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158954e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 809 try 0 launched on executor multi with executor id 875
parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/e712dba9-62ad-405d-8445-bb628a4fb410_calibrate_855910_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/e712dba9-62ad-405d-8445-bb628a4fb410_calibrate_855910_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/5eab55d6-136e-4bea-bcb9-400e62ac6758_characterizeImage_855910_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/5eab55d6-136e-4bea-bcb9-400e62ac6758_characterizeImage_855910_44.stderr
parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5cf542e9a8ec8b771af6236ba238aeb9 with result from task 344
parsl.dataflow.dflow DEBUG: Task 344 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/a18929ed-8e8a-47e6-aeb0-e6b877227cde_calibrate_855910_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/a18929ed-8e8a-47e6-aeb0-e6b877227cde_calibrate_855910_57.stderr
parsl.dataflow.dflow INFO: Task 601 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 799da3ccd41cbcec596990cb04a62a15 with result from task 601
parsl.dataflow.dflow DEBUG: Task 601 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 602 has memoization hash efbf0c2e3e2a1c02637bd9803ef45f2d
parsl.dataflow.memoization INFO: Task 602 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 602 try 0 launched on executor multi with executor id 876
parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/98c45b43-d52b-4bd3-a9fb-ab5f57e809e0_calibrate_855909_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/98c45b43-d52b-4bd3-a9fb-ab5f57e809e0_calibrate_855909_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/c39b822e-04b6-4d49-8069-a41aba7b6c90_characterizeImage_855909_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/c39b822e-04b6-4d49-8069-a41aba7b6c90_characterizeImage_855909_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:44 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 943 try 1 failed
parsl.dataflow.dflow ERROR: Task 943 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 58ecb3603abbc2064411e53d0e773887 with result from task 943
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 944 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 944 try 0 failed
parsl.dataflow.dflow INFO: Task 944 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/9c664b32-a622-4269-85b4-8e16aac3a5a8_calibrate_855911_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/9c664b32-a622-4269-85b4-8e16aac3a5a8_calibrate_855911_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c2cb5987-4744-4a73-a11e-01711df145b9_characterizeImage_855911_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c2cb5987-4744-4a73-a11e-01711df145b9_characterizeImage_855911_8.stderr
parsl.dataflow.dflow INFO: Task 766 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a18a0f97000fb9e83c636b566aa1f9d9 with result from task 766
parsl.dataflow.dflow DEBUG: Task 766 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 767 has memoization hash 0c525f89a8f83ab0440b6a06b0057ee2
parsl.dataflow.memoization INFO: Task 767 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 767 try 0 launched on executor multi with executor id 877
parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6f8a972e-1de8-452c-b9e6-4217141097b6_calibrate_855910_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6f8a972e-1de8-452c-b9e6-4217141097b6_calibrate_855910_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/97fe3d8b-e53d-42d2-85b1-2ad2d8005928_characterizeImage_855910_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/97fe3d8b-e53d-42d2-85b1-2ad2d8005928_characterizeImage_855910_59.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 1072 try 1 failed
parsl.dataflow.dflow ERROR: Task 1072 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 4d110a22fd0f938163de43e6f2161390 with result from task 1072
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 1073 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 1073 try 0 failed
parsl.dataflow.dflow INFO: Task 1073 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/550f25aa-5c82-46ed-8f30-d1707d7c995e_calibrate_855910_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/550f25aa-5c82-46ed-8f30-d1707d7c995e_calibrate_855910_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a0902891-8e8e-4426-96ef-365db6e697c5_characterizeImage_855910_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a0902891-8e8e-4426-96ef-365db6e697c5_characterizeImage_855910_26.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 940 try 1 failed
parsl.dataflow.dflow ERROR: Task 940 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 31dfc87fecdbb2ffc6efb50c17cd23b4 with result from task 940
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 941 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 941 try 0 failed
parsl.dataflow.dflow INFO: Task 941 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/eac2d147-33a0-42d6-8843-3c2bae573dce_calibrate_855911_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/eac2d147-33a0-42d6-8843-3c2bae573dce_calibrate_855911_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/9aa1ef21-e968-49df-89ad-1ceccc2ff409_characterizeImage_855911_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/9aa1ef21-e968-49df-89ad-1ceccc2ff409_characterizeImage_855911_47.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:45 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 682 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9f6b1600b6e9fa9a7b3a13478c5cffca with result from task 682
parsl.dataflow.dflow DEBUG: Task 682 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 683 has memoization hash 5de2266b5b1128188d2d09a3c436bf7f
parsl.dataflow.memoization INFO: Task 683 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15809120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 683 try 0 launched on executor multi with executor id 878
parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/fc2d8795-2bbf-4d69-8144-d455c33bee2e_calibrate_855911_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/fc2d8795-2bbf-4d69-8144-d455c33bee2e_calibrate_855911_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/9f393b75-dbc6-46b2-b267-622feadcc00e_characterizeImage_855911_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/9f393b75-dbc6-46b2-b267-622feadcc00e_characterizeImage_855911_25.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 823 try 1 failed
parsl.dataflow.dflow ERROR: Task 823 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry f14718381899a1619e1cf3f09a95ef33 with result from task 823
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 824 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 824 try 0 failed
parsl.dataflow.dflow INFO: Task 824 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/a0ac4605-36c3-4af0-bcbf-898637dbbed4_calibrate_855911_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/a0ac4605-36c3-4af0-bcbf-898637dbbed4_calibrate_855911_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8af071b6-eb9b-4d32-82ed-ce59eb90ebc7_characterizeImage_855911_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/8af071b6-eb9b-4d32-82ed-ce59eb90ebc7_characterizeImage_855911_20.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:45 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 547 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a24e55ebc0baf166d185a590027f488a with result from task 547
parsl.dataflow.dflow DEBUG: Task 547 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 548 has memoization hash d77328c43dd426a539c03826336d0b07
parsl.dataflow.memoization INFO: Task 548 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 548 try 0 launched on executor multi with executor id 879
parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/45144510-d84f-4061-bd54-9b99a8fdb531_calibrate_855910_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/45144510-d84f-4061-bd54-9b99a8fdb531_calibrate_855910_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/983ad20f-c34f-4fc4-9da3-a6ef678a1ddb_characterizeImage_855910_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/983ad20f-c34f-4fc4-9da3-a6ef678a1ddb_characterizeImage_855910_62.stderr
parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9c7f41f52ab45dd3052e1dba5f6a010b with result from task 367
parsl.dataflow.dflow DEBUG: Task 367 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 368 has memoization hash 6ba5354201a8081028e025b78ae0d622
parsl.dataflow.memoization INFO: Task 368 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 880
parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/5eb16997-8be0-454f-9afe-0051a5dd918f_calibrate_855909_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/5eb16997-8be0-454f-9afe-0051a5dd918f_calibrate_855909_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/703da37d-3fca-4755-b3e3-6aa0828a9ba7_characterizeImage_855909_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/703da37d-3fca-4755-b3e3-6aa0828a9ba7_characterizeImage_855909_38.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:46 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1111 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 669c0ba0f2011f5b009b04ac1aa81983 with result from task 1111
parsl.dataflow.dflow DEBUG: Task 1111 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1112 has memoization hash 887f3a6730a2f11673fb0dfde7f74921
parsl.dataflow.memoization INFO: Task 1112 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1560d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1112 try 0 launched on executor multi with executor id 881
parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/a1a12eb4-0a12-4d74-9026-e3c232666a28_calibrate_855910_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/a1a12eb4-0a12-4d74-9026-e3c232666a28_calibrate_855910_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/8b0b0ebc-5adf-4041-9f8a-12a766517039_characterizeImage_855910_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/8b0b0ebc-5adf-4041-9f8a-12a766517039_characterizeImage_855910_24.stderr
parsl.dataflow.dflow INFO: Task 865 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 686bb50b6884b54c4728ea4bf3cc6d73 with result from task 865
parsl.dataflow.dflow DEBUG: Task 865 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 866 has memoization hash b064d4ad890c1f762a3b34fa1ddc30ba
parsl.dataflow.memoization INFO: Task 866 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 866 try 0 launched on executor multi with executor id 882
parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/c747a1b1-fdc0-44b6-b17d-9eced577e4c3_calibrate_855910_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/c747a1b1-fdc0-44b6-b17d-9eced577e4c3_calibrate_855910_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b1ec9807-3b0b-477f-ab45-6e7e6ba3d354_characterizeImage_855910_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b1ec9807-3b0b-477f-ab45-6e7e6ba3d354_characterizeImage_855910_35.stderr
parsl.dataflow.dflow INFO: Task 1021 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ebe691a9e4fa0961323d3610770dfeb5 with result from task 1021
parsl.dataflow.dflow DEBUG: Task 1021 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1022 has memoization hash f13c826594a182abf6c1745f13ddcffb
parsl.dataflow.memoization INFO: Task 1022 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1022 try 0 launched on executor multi with executor id 883
parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/b2c0b563-0108-4b67-9719-52ed358ece6b_calibrate_855910_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/b2c0b563-0108-4b67-9719-52ed358ece6b_calibrate_855910_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/ae907f24-237b-43f1-93d0-2d392015e674_characterizeImage_855910_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/ae907f24-237b-43f1-93d0-2d392015e674_characterizeImage_855910_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:47 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 955 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3d4868080a7cff73eb491cefd6caba08 with result from task 955
parsl.dataflow.dflow DEBUG: Task 955 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 956 has memoization hash f5f83d77e25938e9a726e9693359c4f2
parsl.dataflow.memoization INFO: Task 956 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 956 try 0 launched on executor multi with executor id 884
parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/1b51178e-7f99-4931-8f2d-d68b3756c0ac_calibrate_855909_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/1b51178e-7f99-4931-8f2d-d68b3756c0ac_calibrate_855909_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/91f97023-ae22-4fb2-8b91-d0d1c119c4db_characterizeImage_855909_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/91f97023-ae22-4fb2-8b91-d0d1c119c4db_characterizeImage_855909_10.stderr
parsl.dataflow.dflow INFO: Task 529 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 68c41bfbfcc03223c40b4a994bb675a2 with result from task 529
parsl.dataflow.dflow DEBUG: Task 529 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 530 has memoization hash e2e58240a1ce8f04c6aed629be26e8a2
parsl.dataflow.memoization INFO: Task 530 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15947100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 530 try 0 launched on executor multi with executor id 885
parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/1ab6df63-02b9-4b71-b934-33d54b68a440_calibrate_855910_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/1ab6df63-02b9-4b71-b934-33d54b68a440_calibrate_855910_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a2c394e6-14a1-464d-bcc5-4504bca96af0_characterizeImage_855910_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a2c394e6-14a1-464d-bcc5-4504bca96af0_characterizeImage_855910_36.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:47 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 515 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cf3a50475e3a0a8129a1b2f655dee89e with result from task 515
parsl.dataflow.dflow DEBUG: Task 515 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/2376b548-b2c9-4ac2-8fd9-7db24fc33abb_calibrate_855910_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/2376b548-b2c9-4ac2-8fd9-7db24fc33abb_calibrate_855910_11.stderr
parsl.dataflow.dflow INFO: Task 487 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f0b387c827d3396f2ea83ddce491dd4e with result from task 487
parsl.dataflow.dflow DEBUG: Task 487 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 488 has memoization hash 91a9a8d8d7027aa9373391adf9267df8
parsl.dataflow.memoization INFO: Task 488 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15944540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 488 try 0 launched on executor multi with executor id 886
parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/15cbc0bc-dc9a-4d12-901d-6165f7619835_calibrate_855909_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/15cbc0bc-dc9a-4d12-901d-6165f7619835_calibrate_855909_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/fef6d108-d03d-454e-835e-e83cd212e5ed_characterizeImage_855909_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/fef6d108-d03d-454e-835e-e83cd212e5ed_characterizeImage_855909_16.stderr
parsl.dataflow.dflow INFO: Task 1012 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f39c1b0279745604638f70de66672693 with result from task 1012
parsl.dataflow.dflow DEBUG: Task 1012 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1013 has memoization hash 4f709b29442fac8ecc67cf513272547d
parsl.dataflow.memoization INFO: Task 1013 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1013 try 0 launched on executor multi with executor id 887
parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/82682231-7401-4da9-bc10-c75a514bb2b5_calibrate_855910_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/82682231-7401-4da9-bc10-c75a514bb2b5_calibrate_855910_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b61b8bb7-c663-42e6-becd-9b2ec397287a_characterizeImage_855910_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/b61b8bb7-c663-42e6-becd-9b2ec397287a_characterizeImage_855910_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:48 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2d86bb2538c7d94f4572378638481385 with result from task 17
parsl.dataflow.dflow DEBUG: Task 17 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/6feb35c8-6162-4b82-8dde-a8f20aef3a01_calibrate_855909_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/6feb35c8-6162-4b82-8dde-a8f20aef3a01_calibrate_855909_45.stderr
parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d8f4373dcdc37acbfdeff85366d9ac1e with result from task 233
parsl.dataflow.dflow DEBUG: Task 233 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/1cf7050f-addf-495d-8b9c-922a03ecd6aa_calibrate_855909_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/1cf7050f-addf-495d-8b9c-922a03ecd6aa_calibrate_855909_26.stderr
parsl.dataflow.dflow INFO: Task 1066 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2b63b3ea86b8cacadcbef96c383d2d9a with result from task 1066
parsl.dataflow.dflow DEBUG: Task 1066 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1067 has memoization hash 5e9ddeea17817f73aa40eff4791987b6
parsl.dataflow.memoization INFO: Task 1067 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1067 try 0 launched on executor multi with executor id 888
parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/b9be2819-92c8-4bf2-9df6-c574f715598d_calibrate_855910_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/b9be2819-92c8-4bf2-9df6-c574f715598d_calibrate_855910_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/2b37cb8b-6722-4cd7-941a-c0f5f7533a61_characterizeImage_855910_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/2b37cb8b-6722-4cd7-941a-c0f5f7533a61_characterizeImage_855910_31.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 847 try 1 failed
parsl.dataflow.dflow ERROR: Task 847 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 8242a98cac56c33208355dd500abf3df with result from task 847
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 848 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 848 try 0 failed
parsl.dataflow.dflow INFO: Task 848 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/b31d7166-22a6-4da9-af76-0f6ee72d6b0f_calibrate_855910_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/b31d7166-22a6-4da9-af76-0f6ee72d6b0f_calibrate_855910_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/f85e8d7d-c248-499a-aa41-99ed2a946c91_characterizeImage_855910_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/f85e8d7d-c248-499a-aa41-99ed2a946c91_characterizeImage_855910_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:48 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1078 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e16c575941579d1b76a8d83bedfce084 with result from task 1078
parsl.dataflow.dflow DEBUG: Task 1078 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1079 has memoization hash 37125ccf9481580a8c8810817e24682a
parsl.dataflow.memoization INFO: Task 1079 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1079 try 0 launched on executor multi with executor id 889
parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/ebead29f-f80d-4a82-a148-4622ec13a647_calibrate_855911_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/ebead29f-f80d-4a82-a148-4622ec13a647_calibrate_855911_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c859df67-1e74-4492-980c-cf360868b857_characterizeImage_855911_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c859df67-1e74-4492-980c-cf360868b857_characterizeImage_855911_9.stderr
parsl.dataflow.dflow INFO: Task 994 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 34ec087cd2d861097b19c3c728dbeb49 with result from task 994
parsl.dataflow.dflow DEBUG: Task 994 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 995 has memoization hash 795c8e9c014014f25edf91247f98867b
parsl.dataflow.memoization INFO: Task 995 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 995 try 0 launched on executor multi with executor id 890
parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/0edb750c-e078-4167-8edb-5314141fafdc_calibrate_855910_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/0edb750c-e078-4167-8edb-5314141fafdc_calibrate_855910_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/81cf190f-6b54-48ff-96e9-87e9e07a6928_characterizeImage_855910_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/81cf190f-6b54-48ff-96e9-87e9e07a6928_characterizeImage_855910_9.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 152 try 0 failed
parsl.dataflow.dflow INFO: Task 152 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/137f21d7-a61d-4a26-953c-bf98c691df7e_calibrate_855911_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/137f21d7-a61d-4a26-953c-bf98c691df7e_calibrate_855911_41.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 79ef392d2714494703d5ab4beb319402
parsl.dataflow.memoization INFO: Task 152 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a425c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 152 try 1 launched on executor multi with executor id 891
parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/137f21d7-a61d-4a26-953c-bf98c691df7e_calibrate_855911_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/137f21d7-a61d-4a26-953c-bf98c691df7e_calibrate_855911_41.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:49 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 586 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ff9d451e097094c06231d369b94b6ddd with result from task 586
parsl.dataflow.dflow DEBUG: Task 586 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 587 has memoization hash 16029cf2744ca4dc98c647dc0128e79a
parsl.dataflow.memoization INFO: Task 587 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 587 try 0 launched on executor multi with executor id 892
parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/6d4625b2-b68f-4f17-830e-f699211f7281_calibrate_855909_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/6d4625b2-b68f-4f17-830e-f699211f7281_calibrate_855909_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/77181560-7c30-4500-84ad-9db9b5388a6d_characterizeImage_855909_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/77181560-7c30-4500-84ad-9db9b5388a6d_characterizeImage_855909_50.stderr
parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 256c2ecc325be0ea77b659422435d5b9 with result from task 40
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:50 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow DEBUG: Task 40 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 52a5fd5e05a1d3880f6e602738325298
parsl.dataflow.memoization INFO: Task 41 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169ff380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 893
parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/92f33fd3-0c59-4962-b23a-a34e2bc231fa_calibrate_855909_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/92f33fd3-0c59-4962-b23a-a34e2bc231fa_calibrate_855909_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/70f5ed46-83e2-42a4-91f4-254084eab382_characterizeImage_855909_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/70f5ed46-83e2-42a4-91f4-254084eab382_characterizeImage_855909_31.stderr
parsl.dataflow.dflow INFO: Task 952 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4d89d0878e104f35f6dbe92d1881e4a3 with result from task 952
parsl.dataflow.dflow DEBUG: Task 952 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 953 has memoization hash 40b57dafdd4264e2cfb9fab0360c3bab
parsl.dataflow.memoization INFO: Task 953 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 953 try 0 launched on executor multi with executor id 894
parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6eafafd0-3bf8-4218-a6bf-aa8970dc0b0a_calibrate_855910_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6eafafd0-3bf8-4218-a6bf-aa8970dc0b0a_calibrate_855910_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/252e616b-69e7-4fae-8213-4661f211ae07_characterizeImage_855910_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/252e616b-69e7-4fae-8213-4661f211ae07_characterizeImage_855910_29.stderr
parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 58020a0853648c5d6ad8e007758a8af2 with result from task 358
parsl.dataflow.dflow DEBUG: Task 358 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 359 has memoization hash 6802c2c1b5ba6ff017cdf5ecfdc53507
parsl.dataflow.memoization INFO: Task 359 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16abbf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 895
parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/f9906f03-8d59-4312-97a9-cbebab84a48a_calibrate_855909_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/f9906f03-8d59-4312-97a9-cbebab84a48a_calibrate_855909_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/94cc707b-1eba-45e3-ac0f-986648eb3d85_characterizeImage_855909_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/94cc707b-1eba-45e3-ac0f-986648eb3d85_characterizeImage_855909_24.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 958 try 0 failed
parsl.dataflow.dflow INFO: Task 958 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/bdd04c08-1946-4778-a3b6-1296bfc4264e_characterizeImage_855910_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/bdd04c08-1946-4778-a3b6-1296bfc4264e_characterizeImage_855910_61.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 958 has memoization hash 7acf68fddc916a95e5ca0642438c3d8d
parsl.dataflow.memoization INFO: Task 958 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 958 try 1 launched on executor multi with executor id 896
parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/bdd04c08-1946-4778-a3b6-1296bfc4264e_characterizeImage_855910_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/bdd04c08-1946-4778-a3b6-1296bfc4264e_characterizeImage_855910_61.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:50 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 448 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2831fd4190ee1972e6626dd7ef2b77a7 with result from task 448
parsl.dataflow.dflow DEBUG: Task 448 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 449 has memoization hash 67b2e718b2e8332a7ba554bc96bad68f
parsl.dataflow.memoization INFO: Task 449 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 449 try 0 launched on executor multi with executor id 897
parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/53649b60-be4c-4ec0-91b9-2e604bcd1a6c_calibrate_855909_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/53649b60-be4c-4ec0-91b9-2e604bcd1a6c_calibrate_855909_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/77e4b1b3-a421-4fdd-bdf7-ae6daab0006d_characterizeImage_855909_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/77e4b1b3-a421-4fdd-bdf7-ae6daab0006d_characterizeImage_855909_47.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:51 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2c663416d7adb2abfdd56c12c4f24d02 with result from task 211
parsl.dataflow.dflow DEBUG: Task 211 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 212 has memoization hash a857bd2ed684b274cf118cdddfd71216
parsl.dataflow.memoization INFO: Task 212 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a66480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 898
parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/2d3dd28d-f2f1-44f1-a9ba-bd86be6169b1_calibrate_855909_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/2d3dd28d-f2f1-44f1-a9ba-bd86be6169b1_calibrate_855909_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/4dc55871-ed7d-4f58-a64a-27716c62a45e_characterizeImage_855909_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/4dc55871-ed7d-4f58-a64a-27716c62a45e_characterizeImage_855909_9.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 980 try 0 failed
parsl.dataflow.dflow INFO: Task 980 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/139674f8-487f-4e75-b947-069d5b995691_calibrate_855911_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/139674f8-487f-4e75-b947-069d5b995691_calibrate_855911_27.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 980 has memoization hash 346228143fc9871a9d1b80cad97ca28d
parsl.dataflow.memoization INFO: Task 980 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 980 try 1 launched on executor multi with executor id 899
parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/139674f8-487f-4e75-b947-069d5b995691_calibrate_855911_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/139674f8-487f-4e75-b947-069d5b995691_calibrate_855911_27.stderr
parsl.dataflow.dflow INFO: Task 455 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 787aabb8e8045467cb69dbe9b1242b31 with result from task 455
parsl.dataflow.dflow DEBUG: Task 455 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/69eea3db-0a8f-4f78-b20b-00ed8a0d2519_calibrate_855910_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/69eea3db-0a8f-4f78-b20b-00ed8a0d2519_calibrate_855910_23.stderr
parsl.dataflow.dflow INFO: Task 613 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b8e6f36aa6b4b7d30e697ddad8b9c5f5 with result from task 613
parsl.dataflow.dflow DEBUG: Task 613 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 614 has memoization hash d41bd213b19e0e5993bd01e6c45c78fe
parsl.dataflow.memoization INFO: Task 614 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 614 try 0 launched on executor multi with executor id 900
parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/7f8c1a62-ed56-4ebe-aefc-a9f9dd203033_calibrate_855910_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/7f8c1a62-ed56-4ebe-aefc-a9f9dd203033_calibrate_855910_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9a44afea-f625-4145-af9b-9f4483525f5e_characterizeImage_855910_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/9a44afea-f625-4145-af9b-9f4483525f5e_characterizeImage_855910_22.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:51 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 638 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f5dbba5a5cfbf9d3db4fdeb5a408a8ba with result from task 638
parsl.dataflow.dflow DEBUG: Task 638 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/db9572b5-8e4e-4a25-a2f9-063ffcddfbde_calibrate_855911_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/db9572b5-8e4e-4a25-a2f9-063ffcddfbde_calibrate_855911_49.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 985 try 1 failed
parsl.dataflow.dflow ERROR: Task 985 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 07e9e47bcfc831eff8f4d5e013c43d01 with result from task 985
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 986 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 986 try 0 failed
parsl.dataflow.dflow INFO: Task 986 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/7b30caab-3e61-4550-9bfe-65e87b568b66_calibrate_855911_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/7b30caab-3e61-4550-9bfe-65e87b568b66_calibrate_855911_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c92cc6f5-cf4f-4d64-9779-b90e3f42d0fc_characterizeImage_855911_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/c92cc6f5-cf4f-4d64-9779-b90e3f42d0fc_characterizeImage_855911_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:52 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:52 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 907 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 99bfb7181ce36dfeee9e52377cd110ac with result from task 907
parsl.dataflow.dflow DEBUG: Task 907 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 908 has memoization hash 0b0eb53b84123c3b53a3cef5bfe22f6e
parsl.dataflow.memoization INFO: Task 908 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156ebc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 908 try 0 launched on executor multi with executor id 901
parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/f7ca1d25-6cd1-4816-acea-0d442c2df65a_calibrate_855910_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/f7ca1d25-6cd1-4816-acea-0d442c2df65a_calibrate_855910_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7c41e0e5-1a0d-4bd6-b22c-9f6e89d0adfe_characterizeImage_855910_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/7c41e0e5-1a0d-4bd6-b22c-9f6e89d0adfe_characterizeImage_855910_54.stderr
parsl.dataflow.dflow INFO: Task 880 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7e3c42ad9a9e03f0d0678ba7c262aba9 with result from task 880
parsl.dataflow.dflow DEBUG: Task 880 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 881 has memoization hash 6d8eee6008bfb4639d4d36ca8da915ee
parsl.dataflow.memoization INFO: Task 881 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156ea020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 881 try 0 launched on executor multi with executor id 902
parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/e6a62c44-1893-4375-980f-338b51b63808_calibrate_855911_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/e6a62c44-1893-4375-980f-338b51b63808_calibrate_855911_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/e5ba6904-f091-4fac-80d6-6804e9224d30_characterizeImage_855911_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855911/e5ba6904-f091-4fac-80d6-6804e9224d30_characterizeImage_855911_36.stderr
parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4acb85c886b42beb887a9239dd63a178 with result from task 37
parsl.dataflow.dflow DEBUG: Task 37 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 3128c04064b386f0d665b6b57ceebaff
parsl.dataflow.memoization INFO: Task 38 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fe200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 903
parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/4c7d7f4a-f1be-495c-abf7-d1bc3e736166_calibrate_855909_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/4c7d7f4a-f1be-495c-abf7-d1bc3e736166_calibrate_855909_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/00256e21-f162-48b0-abac-1faad383dbc9_characterizeImage_855909_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/00256e21-f162-48b0-abac-1faad383dbc9_characterizeImage_855909_34.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:52 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 808f504107b0194da9cb323acb98a022 with result from task 112
parsl.dataflow.dflow DEBUG: Task 112 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 68c2d5f4df76b57eba5575fe5e2edf5a
parsl.dataflow.memoization INFO: Task 113 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 904
parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/857a3d8b-571e-4fba-9359-80fc9e1d4c7b_calibrate_855909_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/857a3d8b-571e-4fba-9359-80fc9e1d4c7b_calibrate_855909_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/2779c069-44b3-4a30-a9fe-fd3b4f67ab16_characterizeImage_855909_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/2779c069-44b3-4a30-a9fe-fd3b4f67ab16_characterizeImage_855909_51.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 110 try 0 failed
parsl.dataflow.dflow INFO: Task 110 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/d398a040-72fb-4773-b23d-96bf076ea2c2_calibrate_855911_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/d398a040-72fb-4773-b23d-96bf076ea2c2_calibrate_855911_59.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 110 has memoization hash b0ae8687c4f7d60c8d45862cd8bc6fec
parsl.dataflow.memoization INFO: Task 110 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 110 try 1 launched on executor multi with executor id 905
parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/d398a040-72fb-4773-b23d-96bf076ea2c2_calibrate_855911_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/d398a040-72fb-4773-b23d-96bf076ea2c2_calibrate_855911_59.stderr
parsl.dataflow.dflow INFO: Task 922 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 82454972e05f907affba19f45f0afb30 with result from task 922
parsl.dataflow.dflow DEBUG: Task 922 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 923 has memoization hash 46584b3691fc2c2ce36ab1788d4e5afe
parsl.dataflow.memoization INFO: Task 923 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15728c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 923 try 0 launched on executor multi with executor id 906
parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/ec8691f7-378b-44c4-9198-7f17773f0433_calibrate_855910_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/ec8691f7-378b-44c4-9198-7f17773f0433_calibrate_855910_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/3c01eca9-482c-46e2-9f65-de2000bf934f_characterizeImage_855910_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/3c01eca9-482c-46e2-9f65-de2000bf934f_characterizeImage_855910_50.stderr
parsl.dataflow.dflow INFO: Task 416 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ad62d2061ea67f2b8978ddf93360b5a7 with result from task 416
parsl.dataflow.dflow DEBUG: Task 416 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/e5ef701f-3613-4c56-bad4-5d3becd57b80_calibrate_855911_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/e5ef701f-3613-4c56-bad4-5d3becd57b80_calibrate_855911_7.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 137 try 1 failed
parsl.dataflow.dflow ERROR: Task 137 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 89b171c4fe9324fbb932a23dd3990938 with result from task 137
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/2a3bf82d-e168-4b5f-a130-e8623b2cfef7_calibrate_855910_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/2a3bf82d-e168-4b5f-a130-e8623b2cfef7_calibrate_855910_4.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:53 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 397 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4330d1e9bdd644c0cfecd4bfb85a7832 with result from task 397
parsl.dataflow.dflow DEBUG: Task 397 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 398 has memoization hash 74efd003d93f9cae2d1ea4764acdbea2
parsl.dataflow.memoization INFO: Task 398 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 907
parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/5fe998ed-146a-4ca7-a5c5-429321432ad4_calibrate_855820_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/5fe998ed-146a-4ca7-a5c5-429321432ad4_calibrate_855820_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/3cc8e705-4823-4be1-b464-b5a98497cfa8_characterizeImage_855820_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/3cc8e705-4823-4be1-b464-b5a98497cfa8_characterizeImage_855820_45.stderr
parsl.dataflow.dflow INFO: Task 1099 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e4db02eaba30ff561f96874924f3f0ae with result from task 1099
parsl.dataflow.dflow DEBUG: Task 1099 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1100 has memoization hash 754f1c05731fe3bff5825022ece9eff3
parsl.dataflow.memoization INFO: Task 1100 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1560c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1100 try 0 launched on executor multi with executor id 908
parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/eb1d3442-c51b-400e-9078-dba7823e7b6d_calibrate_855910_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/eb1d3442-c51b-400e-9078-dba7823e7b6d_calibrate_855910_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a0b55d1a-c76e-448e-8caf-f7efcd1e582e_characterizeImage_855910_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/a0b55d1a-c76e-448e-8caf-f7efcd1e582e_characterizeImage_855910_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:54 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 823b20bd90d4590b1704b4eb8dfd57e2 with result from task 22
parsl.dataflow.dflow DEBUG: Task 22 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 31502b9e9f5a98344a992c4f6c4fff5d
parsl.dataflow.memoization INFO: Task 23 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fdda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 909
parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/5342c5ab-2140-467b-92ab-15270e64fe2f_calibrate_855909_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/5342c5ab-2140-467b-92ab-15270e64fe2f_calibrate_855909_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/48534c7f-0fc6-4cab-8651-2ba90640d703_characterizeImage_855909_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/48534c7f-0fc6-4cab-8651-2ba90640d703_characterizeImage_855909_4.stderr
parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2d7a2101134f7571e7b2e7ff90f73aca with result from task 256
parsl.dataflow.dflow DEBUG: Task 256 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 257 has memoization hash 1ab096999c2f3034f03dff188b245c42
parsl.dataflow.memoization INFO: Task 257 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 910
parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/4b7a2b56-926a-4719-8e17-77ff8d3bc299_calibrate_855820_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/4b7a2b56-926a-4719-8e17-77ff8d3bc299_calibrate_855820_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/3f7cd907-ec3f-4f1e-b09e-390a4ad9fa74_characterizeImage_855820_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/3f7cd907-ec3f-4f1e-b09e-390a4ad9fa74_characterizeImage_855820_33.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:54 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 853 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 36533102723a59e12e32cb8262064078 with result from task 853
parsl.dataflow.dflow DEBUG: Task 853 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 854 has memoization hash a2e729d5ad88ae9d5f3c431c636138bb
parsl.dataflow.memoization INFO: Task 854 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 854 try 0 launched on executor multi with executor id 911
parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/0019dc08-a14c-4fd8-b5c0-6448e19c1538_calibrate_855909_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/0019dc08-a14c-4fd8-b5c0-6448e19c1538_calibrate_855909_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/04860449-e628-494c-b38f-a95dcfbcd802_characterizeImage_855909_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/04860449-e628-494c-b38f-a95dcfbcd802_characterizeImage_855909_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:55 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7f32b4c06bbf3e95f398e95d322f12dd with result from task 275
parsl.dataflow.dflow DEBUG: Task 275 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/b963d7b7-0577-462a-a0b8-16262e565d94_calibrate_855910_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/b963d7b7-0577-462a-a0b8-16262e565d94_calibrate_855910_52.stderr
parsl.dataflow.dflow INFO: Task 883 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 59caf018d23c6979f80fdf674ffdfd96 with result from task 883
parsl.dataflow.dflow DEBUG: Task 883 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 884 has memoization hash db1ca80406d5b8228b6d65a91a6d2c1f
parsl.dataflow.memoization INFO: Task 884 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156ea340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 884 try 0 launched on executor multi with executor id 912
parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/404e9db8-6b7b-487e-aea5-930cb39408ff_calibrate_855910_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/404e9db8-6b7b-487e-aea5-930cb39408ff_calibrate_855910_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/73a09d07-6345-4188-a6bb-5b773b1ac1ea_characterizeImage_855910_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/73a09d07-6345-4188-a6bb-5b773b1ac1ea_characterizeImage_855910_38.stderr
parsl.dataflow.dflow INFO: Task 658 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4dc7df2b7c9a20a70c951e0f5bbfa9a8 with result from task 658
parsl.dataflow.dflow DEBUG: Task 658 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 659 has memoization hash b6800f01a228e5b9095bc20516e3a3b6
parsl.dataflow.memoization INFO: Task 659 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 659 try 0 launched on executor multi with executor id 913
parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/e2df8168-7039-4116-a590-a16d57a43e19_calibrate_855909_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/e2df8168-7039-4116-a590-a16d57a43e19_calibrate_855909_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/db297898-790d-432e-8d2e-d55ac959754d_characterizeImage_855909_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/db297898-790d-432e-8d2e-d55ac959754d_characterizeImage_855909_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:55 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 88a6fefa3b40c60faa771672a0712bb0 with result from task 157
parsl.dataflow.dflow DEBUG: Task 157 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 3a58909f8354fb231e893133c8766672
parsl.dataflow.memoization INFO: Task 158 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a42c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 914
parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/a1b3d2bb-1f45-4a5a-a340-7a95b245fde1_calibrate_855909_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/a1b3d2bb-1f45-4a5a-a340-7a95b245fde1_calibrate_855909_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/52a1e22b-42aa-48c9-b101-0ef92d1dc026_characterizeImage_855909_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/52a1e22b-42aa-48c9-b101-0ef92d1dc026_characterizeImage_855909_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:55 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:56 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:56 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 496 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry db321908b613bd9b2ba56cf29fbc7d17 with result from task 496
parsl.dataflow.dflow DEBUG: Task 496 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 497 has memoization hash 79417711c86f8c8dd2fc59c8e905fb4a
parsl.dataflow.memoization INFO: Task 497 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15944ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 497 try 0 launched on executor multi with executor id 915
parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/6f6e6793-4b80-4621-9311-c67badbdcdf3_calibrate_855909_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/6f6e6793-4b80-4621-9311-c67badbdcdf3_calibrate_855909_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/42ad4c79-eca6-4c87-a81b-9077416fbe9d_characterizeImage_855909_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/42ad4c79-eca6-4c87-a81b-9077416fbe9d_characterizeImage_855909_18.stderr
parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b6b49f24a76c9d7e90f3eb9dca21751d with result from task 169
parsl.dataflow.dflow DEBUG: Task 169 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 170 has memoization hash d7460398fd4d5ffe05578a84a887b15b
parsl.dataflow.memoization INFO: Task 170 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a43880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 916
parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/afa4aaff-f88b-40d2-b750-e8a690ffe052_calibrate_855909_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/afa4aaff-f88b-40d2-b750-e8a690ffe052_calibrate_855909_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/193d03cc-4830-466f-833e-b737dadc3534_characterizeImage_855909_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/193d03cc-4830-466f-833e-b737dadc3534_characterizeImage_855909_43.stderr
parsl.dataflow.dflow INFO: Task 1027 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b8a4bb9a705eb094849c37e9bd360631 with result from task 1027
parsl.dataflow.dflow DEBUG: Task 1027 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1028 has memoization hash d3fa3475141a567da8e08f2e4c5b3215
parsl.dataflow.memoization INFO: Task 1028 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1028 try 0 launched on executor multi with executor id 917
parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/4bf3eda4-16a7-4f3a-8b23-72c6381ea8bc_calibrate_855909_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/4bf3eda4-16a7-4f3a-8b23-72c6381ea8bc_calibrate_855909_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/f2cbd907-26f7-4870-aa5b-2c38f3c42fa5_characterizeImage_855909_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/f2cbd907-26f7-4870-aa5b-2c38f3c42fa5_characterizeImage_855909_15.stderr
parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6eecabf2fbcccec0bc5b18934fad458d with result from task 130
parsl.dataflow.dflow DEBUG: Task 130 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 131 has memoization hash c20312bafa97dd2821fd4753ce76b6c5
parsl.dataflow.memoization INFO: Task 131 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a40fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 918
parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/fb28739e-25bc-4102-9491-ebde27985bb4_calibrate_855909_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/fb28739e-25bc-4102-9491-ebde27985bb4_calibrate_855909_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/babd6655-7ce5-4492-8167-f25fe8eedc2f_characterizeImage_855909_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/babd6655-7ce5-4492-8167-f25fe8eedc2f_characterizeImage_855909_59.stderr
parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bf4c74e444a31c5685442b0b8b56c40f with result from task 296
parsl.dataflow.dflow DEBUG: Task 296 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6911759e-2be1-4ef1-9c82-9b802e45f512_calibrate_855910_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6911759e-2be1-4ef1-9c82-9b802e45f512_calibrate_855910_17.stderr
parsl.dataflow.dflow INFO: Task 1006 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 411574b3c89a389b37bffc547f24cf09 with result from task 1006
parsl.dataflow.dflow DEBUG: Task 1006 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1007 has memoization hash 3f1c8929d2df676f7e2aec2f67532067
parsl.dataflow.memoization INFO: Task 1007 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1007 try 0 launched on executor multi with executor id 919
parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/ff2c9a80-a8b3-43a0-b429-2b2db4cd2384_calibrate_855909_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/ff2c9a80-a8b3-43a0-b429-2b2db4cd2384_calibrate_855909_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/aec76112-4e1f-4dbc-8089-7e8d0f06bb8a_characterizeImage_855909_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/aec76112-4e1f-4dbc-8089-7e8d0f06bb8a_characterizeImage_855909_21.stderr
parsl.dataflow.dflow INFO: Task 778 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e7716551a4f93a433da0d22a36b9737e with result from task 778
parsl.dataflow.dflow DEBUG: Task 778 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 779 has memoization hash 8d910dc61a407e13f6c648bbeffc6a79
parsl.dataflow.memoization INFO: Task 779 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:56:58 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Parsl task 779 try 0 launched on executor multi with executor id 920
parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/e17dbe3c-57be-46bf-8b84-2ad2f19b6751_calibrate_855909_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/e17dbe3c-57be-46bf-8b84-2ad2f19b6751_calibrate_855909_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/b9d511e0-45af-4f43-a4d1-b9b27bba55fc_characterizeImage_855909_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/b9d511e0-45af-4f43-a4d1-b9b27bba55fc_characterizeImage_855909_8.stderr
parsl.dataflow.dflow INFO: Task 494 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fe4e3fd9cf69e4d1c7e90747d271cefb with result from task 494
parsl.dataflow.dflow DEBUG: Task 494 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/15f7ee43-e8e8-49ed-933d-fbf87de3ec7e_calibrate_855911_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/15f7ee43-e8e8-49ed-933d-fbf87de3ec7e_calibrate_855911_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 295
parsl.dataflow.strategy DEBUG: Executor multi has 295 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 20:56:59 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 20:56:59 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 791 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 29dedd432a0174a19795feb0a91e7036 with result from task 791
parsl.dataflow.dflow DEBUG: Task 791 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/af04875e-68da-4810-8e8a-104f32db4360_calibrate_855910_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/af04875e-68da-4810-8e8a-104f32db4360_calibrate_855910_53.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 278 try 0 failed
parsl.dataflow.dflow INFO: Task 278 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/f1f01e01-2819-48d4-97be-cb18f8a0396d_calibrate_855911_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/f1f01e01-2819-48d4-97be-cb18f8a0396d_calibrate_855911_34.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 77418da00e107e98d91f7e21e160d790
parsl.dataflow.memoization INFO: Task 278 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 278 try 1 launched on executor multi with executor id 921
parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/f1f01e01-2819-48d4-97be-cb18f8a0396d_calibrate_855911_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/f1f01e01-2819-48d4-97be-cb18f8a0396d_calibrate_855911_34.stderr
parsl.dataflow.dflow INFO: Task 667 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 78cf8cd67a0167e41241910c3a8c987b with result from task 667
parsl.dataflow.dflow DEBUG: Task 667 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 668 has memoization hash 933a57a581c0e624ad59eaa84995e52c
parsl.dataflow.memoization INFO: Task 668 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15808360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 668 try 0 launched on executor multi with executor id 922
parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/233ffee3-8888-4cca-8c2c-fabf05bb1e3a_calibrate_855909_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/233ffee3-8888-4cca-8c2c-fabf05bb1e3a_calibrate_855909_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/4756df3f-6472-46f7-a6a8-45b5f6e1f99d_characterizeImage_855909_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/4756df3f-6472-46f7-a6a8-45b5f6e1f99d_characterizeImage_855909_33.stderr
parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 09653f709a778a4809b1365d959fef09 with result from task 254
parsl.dataflow.dflow DEBUG: Task 254 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/ed8b204f-f9e0-4fae-8e2a-5436dc4d5298_calibrate_855910_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/ed8b204f-f9e0-4fae-8e2a-5436dc4d5298_calibrate_855910_7.stderr
parsl.dataflow.dflow INFO: Task 961 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8bffe6cf04e0a6c585493ac80d07b880 with result from task 961
parsl.dataflow.dflow DEBUG: Task 961 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 962 has memoization hash 71c8bdf1676e35be1fcc53c14648bd1d
parsl.dataflow.memoization INFO: Task 962 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 962 try 0 launched on executor multi with executor id 923
parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/2209d7fc-9fed-4c05-b1ab-6b1b0bd447d0_calibrate_855909_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/2209d7fc-9fed-4c05-b1ab-6b1b0bd447d0_calibrate_855909_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/47162c81-0123-4f42-afa5-49c1a2f5856b_characterizeImage_855909_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/47162c81-0123-4f42-afa5-49c1a2f5856b_characterizeImage_855909_42.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 146 try 1 failed
parsl.dataflow.dflow ERROR: Task 146 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 676b73370afe2f1ef4f39f09ee6e1c1b with result from task 146
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/d0b16eaa-7653-4e4e-8d2d-6b5e09af82b8_calibrate_855910_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/d0b16eaa-7653-4e4e-8d2d-6b5e09af82b8_calibrate_855910_34.stderr
parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1c07456e656a1da98ecd38606437dc9f with result from task 262
parsl.dataflow.dflow DEBUG: Task 262 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 263 has memoization hash 6ce10c567ea0550f2bb0f35ffe6493dc
parsl.dataflow.memoization INFO: Task 263 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 924
parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/b05e4d11-5f49-421b-968a-e27937f7ee1e_calibrate_855820_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/b05e4d11-5f49-421b-968a-e27937f7ee1e_calibrate_855820_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/4775d0e9-750f-4afd-977d-eac5c6e7559f_characterizeImage_855820_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/4775d0e9-750f-4afd-977d-eac5c6e7559f_characterizeImage_855820_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:00 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:00 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 1024 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b33a5f85182b243afd76e48b673a881b with result from task 1024
parsl.dataflow.dflow DEBUG: Task 1024 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1025 has memoization hash 403b590fc68b893db4c23e7770ed862c
parsl.dataflow.memoization INFO: Task 1025 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1025 try 0 launched on executor multi with executor id 925
parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/1326b4ae-86ab-4a14-b0b6-2c8eafe762e6_calibrate_855909_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/1326b4ae-86ab-4a14-b0b6-2c8eafe762e6_calibrate_855909_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/f0d8d226-e2c1-4d10-9c29-2297e345f592_characterizeImage_855909_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/f0d8d226-e2c1-4d10-9c29-2297e345f592_characterizeImage_855909_19.stderr
parsl.dataflow.dflow INFO: Task 481 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eec4f4f8ca0671b2c4d3b328af5b25ce with result from task 481
parsl.dataflow.dflow DEBUG: Task 481 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 482 has memoization hash db6992591095a1e6ae94fd9fc7b6a691
parsl.dataflow.memoization INFO: Task 482 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 482 try 0 launched on executor multi with executor id 926
parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/e47f955e-899e-483e-82f0-d459c497afc8_calibrate_855909_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/e47f955e-899e-483e-82f0-d459c497afc8_calibrate_855909_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/0e819d5c-7227-4014-918a-76fdab33aba8_characterizeImage_855909_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/0e819d5c-7227-4014-918a-76fdab33aba8_characterizeImage_855909_58.stderr
parsl.dataflow.dflow INFO: Task 410 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d4c2b9e9671296d14a2959b36d7e29e1 with result from task 410
parsl.dataflow.dflow DEBUG: Task 410 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/c5e764df-5046-4c12-8e66-2a74ab581427_calibrate_855911_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/c5e764df-5046-4c12-8e66-2a74ab581427_calibrate_855911_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:00 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 1079 try 0 failed
parsl.dataflow.dflow INFO: Task 1079 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/ebead29f-f80d-4a82-a148-4622ec13a647_calibrate_855911_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/ebead29f-f80d-4a82-a148-4622ec13a647_calibrate_855911_9.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1079 has memoization hash 37125ccf9481580a8c8810817e24682a
parsl.dataflow.memoization INFO: Task 1079 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1079 try 1 launched on executor multi with executor id 927
parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/ebead29f-f80d-4a82-a148-4622ec13a647_calibrate_855911_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/ebead29f-f80d-4a82-a148-4622ec13a647_calibrate_855911_9.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 698 try 0 failed
parsl.dataflow.dflow INFO: Task 698 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/60ccd845-d161-4903-943c-3795d143d14f_calibrate_855910_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/60ccd845-d161-4903-943c-3795d143d14f_calibrate_855910_21.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 698 has memoization hash 9271daf0880bb243ae7a79378bed0c57
parsl.dataflow.memoization INFO: Task 698 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 698 try 1 launched on executor multi with executor id 928
parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/60ccd845-d161-4903-943c-3795d143d14f_calibrate_855910_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/60ccd845-d161-4903-943c-3795d143d14f_calibrate_855910_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:00 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 539 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a6f35acad06134392477b1c2b65f73a5 with result from task 539
parsl.dataflow.dflow DEBUG: Task 539 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/72096477-5c4e-4358-8ece-0747cb3f291f_calibrate_855911_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/72096477-5c4e-4358-8ece-0747cb3f291f_calibrate_855911_51.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:01 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:01 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:01 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 533d28244adb048d0a6ae4cfa5041b14 with result from task 155
parsl.dataflow.dflow DEBUG: Task 155 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/608b3a90-8ab2-4965-a13e-3da6418aaf24_calibrate_855911_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/608b3a90-8ab2-4965-a13e-3da6418aaf24_calibrate_855911_37.stderr
parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e5e2759a0a14e5ad555a76dd67b7ecb6 with result from task 350
parsl.dataflow.dflow DEBUG: Task 350 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/7626a4f6-e012-44fa-88fa-98a1ac995913_calibrate_855909_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/7626a4f6-e012-44fa-88fa-98a1ac995913_calibrate_855909_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:02 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 1004 try 1 failed
parsl.dataflow.dflow ERROR: Task 1004 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry adf6cb0f74bf726d49569545af5a744c with result from task 1004
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/c0b60462-6b9e-4161-9fe1-300e9185d26d_calibrate_855911_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/c0b60462-6b9e-4161-9fe1-300e9185d26d_calibrate_855911_32.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:02 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 1022 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f13c826594a182abf6c1745f13ddcffb with result from task 1022
parsl.dataflow.dflow DEBUG: Task 1022 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/b2c0b563-0108-4b67-9719-52ed358ece6b_calibrate_855910_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/b2c0b563-0108-4b67-9719-52ed358ece6b_calibrate_855910_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:02 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 619 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1a18198a4f4c3a99bf02b7fec85b3de8 with result from task 619
parsl.dataflow.dflow DEBUG: Task 619 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 620 has memoization hash 504c2bd0f04f626b71731df74e3a1f7e
parsl.dataflow.memoization INFO: Task 620 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 620 try 0 launched on executor multi with executor id 929
parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/33190932-3f3c-407f-8c9d-17f84c38b814_calibrate_855909_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/33190932-3f3c-407f-8c9d-17f84c38b814_calibrate_855909_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/1e8ac2dc-3a26-4bea-9767-d7aa40c2b94b_characterizeImage_855909_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/1e8ac2dc-3a26-4bea-9767-d7aa40c2b94b_characterizeImage_855909_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:03 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 293 try 1 failed
parsl.dataflow.dflow ERROR: Task 293 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry c1340757feaff4f75202df85c1400c08 with result from task 293
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/1e37b574-52b0-418c-a9db-600752e5554b_calibrate_855911_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/1e37b574-52b0-418c-a9db-600752e5554b_calibrate_855911_14.stderr
parsl.dataflow.dflow INFO: Task 1039 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ee1f3e88476314832ac87aa22baf5f99 with result from task 1039
parsl.dataflow.dflow DEBUG: Task 1039 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1040 has memoization hash 0e789652a1c39b090b34aaa0c44dbc64
parsl.dataflow.memoization INFO: Task 1040 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1040 try 0 launched on executor multi with executor id 930
parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/f408d0d8-18b1-4eaa-9626-c36503685bc3_calibrate_855820_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/f408d0d8-18b1-4eaa-9626-c36503685bc3_calibrate_855820_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/043c5258-3f02-4bcd-a957-9e064c89d9b5_characterizeImage_855820_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/043c5258-3f02-4bcd-a957-9e064c89d9b5_characterizeImage_855820_39.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:03 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 731 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3901c9562a0568e9cc2ef3c73d5d083e with result from task 731
parsl.dataflow.dflow DEBUG: Task 731 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/c5fd6e4d-3831-46c8-a994-9b95d770ab26_calibrate_855911_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/c5fd6e4d-3831-46c8-a994-9b95d770ab26_calibrate_855911_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:03 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 60aba6e9537b6d4dedd336405005ba02 with result from task 337
parsl.dataflow.dflow DEBUG: Task 337 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 338 has memoization hash e252698d54ee76437c0fb16b5cfa14fc
parsl.dataflow.memoization INFO: Task 338 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16aba840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 931
parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7388943f-22af-4226-b461-839a5b39d052_calibrate_855820_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7388943f-22af-4226-b461-839a5b39d052_calibrate_855820_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/4c5337f9-6458-446a-bce2-ca968e7d705b_characterizeImage_855820_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/4c5337f9-6458-446a-bce2-ca968e7d705b_characterizeImage_855820_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:04 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 859 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 52479442d118ec5e09fbd1e759df2106 with result from task 859
parsl.dataflow.dflow DEBUG: Task 859 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 860 has memoization hash 1fe529a12dc22844074b738714fcd222
parsl.dataflow.memoization INFO: Task 860 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 860 try 0 launched on executor multi with executor id 932
parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/77bc71e5-9fde-453e-8292-03e6e4f6cc5e_calibrate_855909_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/77bc71e5-9fde-453e-8292-03e6e4f6cc5e_calibrate_855909_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/6e2e5b54-4452-46b0-a1e7-b04edf11655a_characterizeImage_855909_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/6e2e5b54-4452-46b0-a1e7-b04edf11655a_characterizeImage_855909_35.stderr
parsl.dataflow.dflow INFO: Task 899 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9d05fbcb10181f8470d48c0c3c1ac35d with result from task 899
parsl.dataflow.dflow DEBUG: Task 899 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6c3131ac-dbeb-4634-8000-50c33395ac1d_calibrate_855910_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6c3131ac-dbeb-4634-8000-50c33395ac1d_calibrate_855910_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:04 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 443 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7038937d9c07bbb992cd3b0036148455 with result from task 443
parsl.dataflow.dflow DEBUG: Task 443 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6cf5d066-ac35-414d-84f4-66d62329c8a4_calibrate_855910_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6cf5d066-ac35-414d-84f4-66d62329c8a4_calibrate_855910_60.stderr
parsl.dataflow.dflow INFO: Task 821 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c2853a0c0390f71053fa2803f3a55208 with result from task 821
parsl.dataflow.dflow DEBUG: Task 821 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/874038c0-6129-4fb1-a3a4-1dad17c3ca65_calibrate_855911_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/874038c0-6129-4fb1-a3a4-1dad17c3ca65_calibrate_855911_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:05 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 850 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bf770d3689c08d0791ad7139e4c29c49 with result from task 850
parsl.dataflow.dflow DEBUG: Task 850 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 851 has memoization hash 35348700254419472a65f4c11bf3f6a5
parsl.dataflow.memoization INFO: Task 851 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 851 try 0 launched on executor multi with executor id 933
parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/5349dcb1-0784-41ce-a709-ef204397be47_calibrate_855909_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/5349dcb1-0784-41ce-a709-ef204397be47_calibrate_855909_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/11a08ca0-e0d9-4786-9d91-19aed048bf37_characterizeImage_855909_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/11a08ca0-e0d9-4786-9d91-19aed048bf37_characterizeImage_855909_13.stderr
parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3b916828894c37f12d7107835046a03f with result from task 272
parsl.dataflow.dflow DEBUG: Task 272 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/aa1093dd-173c-4b83-bdde-3cc85224f5d1_calibrate_855909_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/aa1093dd-173c-4b83-bdde-3cc85224f5d1_calibrate_855909_7.stderr
parsl.dataflow.dflow INFO: Task 703 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 244c05ef383116b5b0934e69e10928ef with result from task 703
parsl.dataflow.dflow DEBUG: Task 703 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 704 has memoization hash 5983366c8a670ddfa1b8fc722ee4cc67
parsl.dataflow.memoization INFO: Task 704 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 704 try 0 launched on executor multi with executor id 934
parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/59fcef67-c828-4464-9924-dc389ff4092d_calibrate_855909_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/59fcef67-c828-4464-9924-dc389ff4092d_calibrate_855909_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/416f3cfe-711b-4d65-bc3c-7c415e030bfd_characterizeImage_855909_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/416f3cfe-711b-4d65-bc3c-7c415e030bfd_characterizeImage_855909_27.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:05 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:06 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 559 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 370fdfc9d61c5013360ffd6c9761c36c with result from task 559
parsl.dataflow.dflow DEBUG: Task 559 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 560 has memoization hash f382ed284587e8d3ee313365fe01350c
parsl.dataflow.memoization INFO: Task 560 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 560 try 0 launched on executor multi with executor id 935
parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/97c9bf25-35b5-4a56-8ae6-6cb62bf5672d_calibrate_855909_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/97c9bf25-35b5-4a56-8ae6-6cb62bf5672d_calibrate_855909_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/efaa552c-ea8a-4119-aaaf-07272407d8f1_characterizeImage_855909_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/efaa552c-ea8a-4119-aaaf-07272407d8f1_characterizeImage_855909_53.stderr
parsl.dataflow.dflow INFO: Task 437 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c267caf8d08b8b601359b90b2f913e6d with result from task 437
parsl.dataflow.dflow DEBUG: Task 437 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/b5cf2999-9fc6-440a-87e9-830137771018_calibrate_855911_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/b5cf2999-9fc6-440a-87e9-830137771018_calibrate_855911_60.stderr
parsl.dataflow.dflow INFO: Task 1058 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3b6af49496c4d80760fa289ee6245d99 with result from task 1058
parsl.dataflow.dflow DEBUG: Task 1058 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/9951a541-ed1c-4dd4-8424-4bb33b7a3c6c_calibrate_855910_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/9951a541-ed1c-4dd4-8424-4bb33b7a3c6c_calibrate_855910_48.stderr
parsl.dataflow.dflow INFO: Task 931 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6562a1fe08263739b9edf44ee3ea57c6 with result from task 931
parsl.dataflow.dflow DEBUG: Task 931 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 932 has memoization hash d25546466483eb3d9532705bc1958759
parsl.dataflow.memoization INFO: Task 932 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15729580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 932 try 0 launched on executor multi with executor id 936
parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/fed42ed8-966a-47db-8904-197fe07539cf_calibrate_855909_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/fed42ed8-966a-47db-8904-197fe07539cf_calibrate_855909_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/f280d8c3-0982-4f29-a8ef-cc735bc3c4cc_characterizeImage_855909_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/f280d8c3-0982-4f29-a8ef-cc735bc3c4cc_characterizeImage_855909_32.stderr
parsl.dataflow.dflow INFO: Task 500 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c41c62e11b12a82d284db429b68a5141 with result from task 500
parsl.dataflow.dflow DEBUG: Task 500 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/27548139-64d6-467b-b22c-bf3d8a966312_calibrate_855909_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/27548139-64d6-467b-b22c-bf3d8a966312_calibrate_855909_20.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:06 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 683 try 0 failed
parsl.dataflow.dflow INFO: Task 683 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/fc2d8795-2bbf-4d69-8144-d455c33bee2e_calibrate_855911_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/fc2d8795-2bbf-4d69-8144-d455c33bee2e_calibrate_855911_25.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 683 has memoization hash 5de2266b5b1128188d2d09a3c436bf7f
parsl.dataflow.memoization INFO: Task 683 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15809120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 683 try 1 launched on executor multi with executor id 937
parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/fc2d8795-2bbf-4d69-8144-d455c33bee2e_calibrate_855911_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/fc2d8795-2bbf-4d69-8144-d455c33bee2e_calibrate_855911_25.stderr
parsl.dataflow.dflow INFO: Task 761 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 69e64f2d7c181fc6cc02920d5cfaa61a with result from task 761
parsl.dataflow.dflow DEBUG: Task 761 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/b385a0f6-322f-47c8-b109-0f442c42fa5e_calibrate_855910_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/b385a0f6-322f-47c8-b109-0f442c42fa5e_calibrate_855910_30.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 1001 try 1 failed
parsl.dataflow.dflow ERROR: Task 1001 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 7cabc7b4f1da8c753a2e3846d3a95ba2 with result from task 1001
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/851e6f2e-f498-4375-ba09-41b714b0c36b_calibrate_855911_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/851e6f2e-f498-4375-ba09-41b714b0c36b_calibrate_855911_35.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:07 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0a83697838542b4dce18be5a90fb11ca with result from task 106
parsl.dataflow.dflow DEBUG: Task 106 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 317c8cf672a9576dac277963d932a0ae
parsl.dataflow.memoization INFO: Task 107 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 938
parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/3156187d-b017-45f9-ab8b-a9056fed69d5_calibrate_855909_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/3156187d-b017-45f9-ab8b-a9056fed69d5_calibrate_855909_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/e19b6db5-756d-4b82-88ce-7955999a4c81_characterizeImage_855909_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/e19b6db5-756d-4b82-88ce-7955999a4c81_characterizeImage_855909_49.stderr
parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fd9831a1355c61e6fdd07eed301095d6 with result from task 163
parsl.dataflow.dflow DEBUG: Task 163 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 7adcf9beece119f1af2bf82d13afc106
parsl.dataflow.memoization INFO: Task 164 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a43240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 939
parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/34155c04-de53-4a83-b402-959b4958ddf7_calibrate_855820_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/34155c04-de53-4a83-b402-959b4958ddf7_calibrate_855820_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/3719b576-79ad-4b8a-aefe-850d3b867ec6_characterizeImage_855820_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/3719b576-79ad-4b8a-aefe-850d3b867ec6_characterizeImage_855820_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:07 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 964 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f518ef0c50dee16be2ee8528da7a6598 with result from task 964
parsl.dataflow.dflow DEBUG: Task 964 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 965 has memoization hash 870792b8bfb47a0e6fb22e1754846b0a
parsl.dataflow.memoization INFO: Task 965 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 965 try 0 launched on executor multi with executor id 940
parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/14f395a7-a40f-4b36-b609-b1ec9acc5ed8_calibrate_855909_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/14f395a7-a40f-4b36-b609-b1ec9acc5ed8_calibrate_855909_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/d2f137b1-2ecd-4710-a401-ff67c98329ec_characterizeImage_855909_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/d2f137b1-2ecd-4710-a401-ff67c98329ec_characterizeImage_855909_40.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:08 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:09 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6e8fabcdc812273db85f53adcc588782 with result from task 35
parsl.dataflow.dflow DEBUG: Task 35 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/f9e3a686-e7de-482f-be9b-3537a6bb7fed_calibrate_855910_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/f9e3a686-e7de-482f-be9b-3537a6bb7fed_calibrate_855910_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:09 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5c95d1101c9012259ef8b60daf5c016c with result from task 95
parsl.dataflow.dflow DEBUG: Task 95 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/9209a414-c040-4101-9a58-82fd120b8de4_calibrate_855910_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/9209a414-c040-4101-9a58-82fd120b8de4_calibrate_855910_55.stderr
parsl.dataflow.dflow INFO: Task 460 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7b782907e42e21e0ead315ad87f57d71 with result from task 460
parsl.dataflow.dflow DEBUG: Task 460 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 461 has memoization hash 664a4c4f975c8815a0eb87bae9c7451b
parsl.dataflow.memoization INFO: Task 461 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 461 try 0 launched on executor multi with executor id 941
parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/068475ad-76f2-44f1-99b6-c862f396f8e9_calibrate_855820_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/068475ad-76f2-44f1-99b6-c862f396f8e9_calibrate_855820_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/4152262c-c5ad-42f2-a5c9-fa5bbc820ef0_characterizeImage_855820_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/4152262c-c5ad-42f2-a5c9-fa5bbc820ef0_characterizeImage_855820_15.stderr
parsl.dataflow.dflow INFO: Task 799 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dffcc47853e66040cbb0f8c406308654 with result from task 799
parsl.dataflow.dflow DEBUG: Task 799 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 800 has memoization hash 56b7ffed9adf07c82f988fcecb394772
parsl.dataflow.memoization INFO: Task 800 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15894b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 800 try 0 launched on executor multi with executor id 942
parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/49fe7c1e-b023-4d43-aa87-2f716d5a7b39_calibrate_855909_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/49fe7c1e-b023-4d43-aa87-2f716d5a7b39_calibrate_855909_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/8083e6ed-55c2-4346-bb73-e777b6e2ee60_characterizeImage_855909_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/8083e6ed-55c2-4346-bb73-e777b6e2ee60_characterizeImage_855909_3.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:10 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 553 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 59d9135ca65e6617b453930184cae62e with result from task 553
parsl.dataflow.dflow DEBUG: Task 553 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 554 has memoization hash 0df99efcb55612b6b2ff420a45071ea4
parsl.dataflow.memoization INFO: Task 554 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 554 try 0 launched on executor multi with executor id 943
parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/a1441880-e357-413c-a1c3-831e0383bbb0_calibrate_855909_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/a1441880-e357-413c-a1c3-831e0383bbb0_calibrate_855909_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/d25760e0-7889-4b59-891c-3f7b383a6659_characterizeImage_855909_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/d25760e0-7889-4b59-891c-3f7b383a6659_characterizeImage_855909_48.stderr
parsl.dataflow.dflow INFO: Task 1102 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 60efd1fff1d693dab1cabc7979677329 with result from task 1102
parsl.dataflow.dflow DEBUG: Task 1102 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1103 has memoization hash 4edf0279be1c8d675d1a3dbfe0050aa5
parsl.dataflow.memoization INFO: Task 1103 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1560c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1103 try 0 launched on executor multi with executor id 944
parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/317684b9-714b-48e6-941d-3b739e500022_calibrate_855820_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/317684b9-714b-48e6-941d-3b739e500022_calibrate_855820_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/5d35ab60-acd1-4811-8b9d-42f9098c6a10_characterizeImage_855820_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/5d35ab60-acd1-4811-8b9d-42f9098c6a10_characterizeImage_855820_51.stderr
parsl.dataflow.dflow INFO: Task 634 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b1b059b8d83e096da5a6a5af2b3f73fa with result from task 634
parsl.dataflow.dflow DEBUG: Task 634 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 635 has memoization hash 24ac138105e2f1f132aaea48c2376faf
parsl.dataflow.memoization INFO: Task 635 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:10 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Parsl task 635 try 0 launched on executor multi with executor id 945
parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/9ca73e40-4427-4446-87ba-7850aee51b63_calibrate_855820_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/9ca73e40-4427-4446-87ba-7850aee51b63_calibrate_855820_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ad2d3277-f757-4b87-b4ba-9ac08ea5adfd_characterizeImage_855820_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ad2d3277-f757-4b87-b4ba-9ac08ea5adfd_characterizeImage_855820_19.stderr
parsl.dataflow.dflow INFO: Task 953 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 40b57dafdd4264e2cfb9fab0360c3bab with result from task 953
parsl.dataflow.dflow DEBUG: Task 953 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6eafafd0-3bf8-4218-a6bf-aa8970dc0b0a_calibrate_855910_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6eafafd0-3bf8-4218-a6bf-aa8970dc0b0a_calibrate_855910_29.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:11 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 1015 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4688c2bb8421a430df0b203449375278 with result from task 1015
parsl.dataflow.dflow DEBUG: Task 1015 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1016 has memoization hash 2a499d40ead93cc26fab893e6b404a77
parsl.dataflow.memoization INFO: Task 1016 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1016 try 0 launched on executor multi with executor id 946
parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/2cb92a27-c5da-4f07-b252-0b6082130fbf_calibrate_855909_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/2cb92a27-c5da-4f07-b252-0b6082130fbf_calibrate_855909_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/658ecdf0-5d3c-411b-b98e-69b32732774e_characterizeImage_855909_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/658ecdf0-5d3c-411b-b98e-69b32732774e_characterizeImage_855909_6.stderr
parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1fb59a25919fa162e3c017e079579c65 with result from task 10
parsl.dataflow.dflow DEBUG: Task 10 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 11 has memoization hash c8877a61dcaad6f463823d6e9ea9c84d
parsl.dataflow.memoization INFO: Task 11 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fcea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 947
parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/e59b8adf-cc32-4181-bade-8b5a921f4fcf_calibrate_855820_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/e59b8adf-cc32-4181-bade-8b5a921f4fcf_calibrate_855820_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/aee9a4a4-2cb4-4a22-8cfd-1defc1fb950e_characterizeImage_855820_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/aee9a4a4-2cb4-4a22-8cfd-1defc1fb950e_characterizeImage_855820_32.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 995 try 0 failed
parsl.dataflow.dflow INFO: Task 995 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/0edb750c-e078-4167-8edb-5314141fafdc_calibrate_855910_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/0edb750c-e078-4167-8edb-5314141fafdc_calibrate_855910_9.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 995 has memoization hash 795c8e9c014014f25edf91247f98867b
parsl.dataflow.memoization INFO: Task 995 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 995 try 1 launched on executor multi with executor id 948
parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/0edb750c-e078-4167-8edb-5314141fafdc_calibrate_855910_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/0edb750c-e078-4167-8edb-5314141fafdc_calibrate_855910_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:11 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 1067 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5e9ddeea17817f73aa40eff4791987b6 with result from task 1067
parsl.dataflow.dflow DEBUG: Task 1067 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/b9be2819-92c8-4bf2-9df6-c574f715598d_calibrate_855910_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/b9be2819-92c8-4bf2-9df6-c574f715598d_calibrate_855910_31.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:12 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b6f2a7d11ff911870720f39b20715039 with result from task 268
parsl.dataflow.dflow DEBUG: Task 268 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 269 has memoization hash 410d98db872ea9d281e0c325151f1f69
parsl.dataflow.memoization INFO: Task 269 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 949
parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7fad7af5-84c8-4d41-8855-9542048ebf8b_calibrate_855820_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7fad7af5-84c8-4d41-8855-9542048ebf8b_calibrate_855820_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/dce35c21-7d97-4e16-ad95-28d61e8f3a90_characterizeImage_855820_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/dce35c21-7d97-4e16-ad95-28d61e8f3a90_characterizeImage_855820_55.stderr
parsl.dataflow.dflow INFO: Task 886 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3cf7610271a5d288ac714317d3900ddc with result from task 886
parsl.dataflow.dflow DEBUG: Task 886 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 887 has memoization hash e5409d79154942472868d3ae3e80f1a9
parsl.dataflow.memoization INFO: Task 887 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156ea660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 887 try 0 launched on executor multi with executor id 950
parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/dc9ff95b-7ae9-43f8-b845-187b6d1cfa13_calibrate_855909_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/dc9ff95b-7ae9-43f8-b845-187b6d1cfa13_calibrate_855909_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/212764aa-eb57-400c-b873-c889eec56055_characterizeImage_855909_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/212764aa-eb57-400c-b873-c889eec56055_characterizeImage_855909_1.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:12 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.dataflow.dflow INFO: Task 389 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2572e5fb14491cb6acdb24c132f79f8f with result from task 389
parsl.dataflow.dflow DEBUG: Task 389 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/f4305e01-5bd9-4b9d-9ae6-dfa67182e148_calibrate_855909_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/f4305e01-5bd9-4b9d-9ae6-dfa67182e148_calibrate_855909_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:13 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.dataflow.dflow INFO: Task 835 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8103ff89eb5dc532afae57343911b635 with result from task 835
parsl.dataflow.dflow DEBUG: Task 835 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 836 has memoization hash a74b33752a9e5f4265e2b2edd12924fc
parsl.dataflow.memoization INFO: Task 836 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15897100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 836 try 0 launched on executor multi with executor id 951
parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/169b8ff4-c9f5-467f-bccc-39140eadfe93_calibrate_855909_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/169b8ff4-c9f5-467f-bccc-39140eadfe93_calibrate_855909_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/9a8f8d87-b354-4593-9dfa-27d9f780b288_characterizeImage_855909_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/9a8f8d87-b354-4593-9dfa-27d9f780b288_characterizeImage_855909_57.stderr
parsl.dataflow.dflow INFO: Task 983 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 72e458a4b2c2b3230805bc2c5b56b32b with result from task 983
parsl.dataflow.dflow DEBUG: Task 983 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/c6967517-8e3b-44c1-87d3-e744febb606c_calibrate_855911_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/c6967517-8e3b-44c1-87d3-e744febb606c_calibrate_855911_30.stderr
parsl.dataflow.dflow INFO: Task 677 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8c9fb1ff44683cb100b5b6f724b758bf with result from task 677
parsl.dataflow.dflow DEBUG: Task 677 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/4799e5c2-0f26-4c4f-b5cf-7a334fd6e5e3_calibrate_855911_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/4799e5c2-0f26-4c4f-b5cf-7a334fd6e5e3_calibrate_855911_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:14 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 980 try 1 failed
parsl.dataflow.dflow ERROR: Task 980 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 346228143fc9871a9d1b80cad97ca28d with result from task 980
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/139674f8-487f-4e75-b947-069d5b995691_calibrate_855911_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/139674f8-487f-4e75-b947-069d5b995691_calibrate_855911_27.stderr
parsl.dataflow.dflow INFO: Task 917 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2032390fe3805092b9d95881a0359710 with result from task 917
parsl.dataflow.dflow DEBUG: Task 917 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/ca403cb0-5807-46b9-b676-df869a553e85_calibrate_855910_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/ca403cb0-5807-46b9-b676-df869a553e85_calibrate_855910_6.stderr
parsl.dataflow.dflow INFO: Task 743 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7e286da1999a4d684b02d375db7f15bc with result from task 743
parsl.dataflow.dflow DEBUG: Task 743 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6aafe7b7-5807-48e7-a052-46ad4af88d13_calibrate_855910_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6aafe7b7-5807-48e7-a052-46ad4af88d13_calibrate_855910_3.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 110 try 1 failed
parsl.dataflow.dflow ERROR: Task 110 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry b0ae8687c4f7d60c8d45862cd8bc6fec with result from task 110
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/d398a040-72fb-4773-b23d-96bf076ea2c2_calibrate_855911_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/d398a040-72fb-4773-b23d-96bf076ea2c2_calibrate_855911_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:14 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.dataflow.dflow INFO: Task 614 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d41bd213b19e0e5993bd01e6c45c78fe with result from task 614
parsl.dataflow.dflow DEBUG: Task 614 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/7f8c1a62-ed56-4ebe-aefc-a9f9dd203033_calibrate_855910_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/7f8c1a62-ed56-4ebe-aefc-a9f9dd203033_calibrate_855910_22.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 698 try 1 failed
parsl.dataflow.dflow ERROR: Task 698 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 9271daf0880bb243ae7a79378bed0c57 with result from task 698
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/60ccd845-d161-4903-943c-3795d143d14f_calibrate_855910_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/60ccd845-d161-4903-943c-3795d143d14f_calibrate_855910_21.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 866 try 0 failed
parsl.dataflow.dflow INFO: Task 866 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/c747a1b1-fdc0-44b6-b17d-9eced577e4c3_calibrate_855910_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/c747a1b1-fdc0-44b6-b17d-9eced577e4c3_calibrate_855910_35.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 866 has memoization hash b064d4ad890c1f762a3b34fa1ddc30ba
parsl.dataflow.memoization INFO: Task 866 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 866 try 1 launched on executor multi with executor id 952
parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/c747a1b1-fdc0-44b6-b17d-9eced577e4c3_calibrate_855910_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/c747a1b1-fdc0-44b6-b17d-9eced577e4c3_calibrate_855910_35.stderr
parsl.dataflow.dflow INFO: Task 527 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dba7e236a38194c720b49c96aa30b216 with result from task 527
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:15 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.dataflow.dflow DEBUG: Task 527 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/6aab1a49-b9a9-4326-bb5e-515cc7609091_calibrate_855911_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/6aab1a49-b9a9-4326-bb5e-515cc7609091_calibrate_855911_52.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 958 try 1 failed
parsl.dataflow.dflow ERROR: Task 958 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 7acf68fddc916a95e5ca0642438c3d8d with result from task 958
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 959 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 959 try 0 failed
parsl.dataflow.dflow INFO: Task 959 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/bbaef560-191d-4d71-b5f9-6f2a0ab1de37_calibrate_855910_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/bbaef560-191d-4d71-b5f9-6f2a0ab1de37_calibrate_855910_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/bdd04c08-1946-4778-a3b6-1296bfc4264e_characterizeImage_855910_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855910/bdd04c08-1946-4778-a3b6-1296bfc4264e_characterizeImage_855910_61.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:15 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:16 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 568 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 55b894b39f80d7c716557491f30be694 with result from task 568
parsl.dataflow.dflow DEBUG: Task 568 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 569 has memoization hash 9e1dcee26d7e769f187bc7e461b7a3d6
parsl.dataflow.memoization INFO: Task 569 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 569 try 0 launched on executor multi with executor id 953
parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7f50291a-a786-4033-a024-b5ab0f53255b_calibrate_855820_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7f50291a-a786-4033-a024-b5ab0f53255b_calibrate_855820_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/7886e349-ffb6-4da2-83fe-b94e2d0dd6c7_characterizeImage_855820_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/7886e349-ffb6-4da2-83fe-b94e2d0dd6c7_characterizeImage_855820_6.stderr
parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b455011bf398a6bef8474edf9ba492d5 with result from task 290
parsl.dataflow.dflow DEBUG: Task 290 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/43e331d5-226b-4588-b778-ffd157acd4f4_calibrate_855909_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/43e331d5-226b-4588-b778-ffd157acd4f4_calibrate_855909_11.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:16 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 1109 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9cfc81dfab522799a2ca86b80fbc6050 with result from task 1109
parsl.dataflow.dflow DEBUG: Task 1109 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/8bfb21c8-cdf9-4f9b-9cb9-873ef10d015c_calibrate_855911_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/8bfb21c8-cdf9-4f9b-9cb9-873ef10d015c_calibrate_855911_56.stderr
parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 019018ca9c18eeec61b7fb7feb9bdc53 with result from task 92
parsl.dataflow.dflow DEBUG: Task 92 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/82fd2d5a-8b6a-4e79-aa92-6b90c754d4c2_calibrate_855910_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/82fd2d5a-8b6a-4e79-aa92-6b90c754d4c2_calibrate_855910_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:17 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:17 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6d4cbd81c5d64840794f7834c06931c4 with result from task 248
parsl.dataflow.dflow DEBUG: Task 248 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/f61911c7-9f60-4ace-aead-fb06c6fd42fb_calibrate_855909_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/f61911c7-9f60-4ace-aead-fb06c6fd42fb_calibrate_855909_29.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:18 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 745 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 55666f5781a91b008ccc5454f21c0658 with result from task 745
parsl.dataflow.dflow DEBUG: Task 745 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 746 has memoization hash 2a8a40f5b5b07ef99dbf7ba1d80f69d7
parsl.dataflow.memoization INFO: Task 746 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15859300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 746 try 0 launched on executor multi with executor id 954
parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/7419e133-b980-4e4e-82de-f7509b677834_calibrate_855909_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/7419e133-b980-4e4e-82de-f7509b677834_calibrate_855909_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/e7e5ef6d-6383-4cff-986a-abc4c3dc521b_characterizeImage_855909_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/e7e5ef6d-6383-4cff-986a-abc4c3dc521b_characterizeImage_855909_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:19 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry df9178c0deb6bf91cc200e03d110ca01 with result from task 320
parsl.dataflow.dflow DEBUG: Task 320 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/7f84eb19-d9b2-471b-a039-839c1b925310_calibrate_855909_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/7f84eb19-d9b2-471b-a039-839c1b925310_calibrate_855909_36.stderr
parsl.dataflow.dflow INFO: Task 767 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0c525f89a8f83ab0440b6a06b0057ee2 with result from task 767
parsl.dataflow.dflow DEBUG: Task 767 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6f8a972e-1de8-452c-b9e6-4217141097b6_calibrate_855910_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/6f8a972e-1de8-452c-b9e6-4217141097b6_calibrate_855910_59.stderr
parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5e45d4882a04a974c1a6b45d6fe6571c with result from task 284
parsl.dataflow.dflow DEBUG: Task 284 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/290dc026-69b5-4843-b2e5-5d2c98504474_calibrate_855910_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/290dc026-69b5-4843-b2e5-5d2c98504474_calibrate_855910_41.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:19 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 758 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9e2c0686fae8f327dc644dc3b9a880bf with result from task 758
parsl.dataflow.dflow DEBUG: Task 758 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/c59ed363-f5be-443f-86d8-4cad85bbdf47_calibrate_855910_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/c59ed363-f5be-443f-86d8-4cad85bbdf47_calibrate_855910_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:20 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 782 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 82d9183aab6665929154bcb9ce61b1e4 with result from task 782
parsl.dataflow.dflow DEBUG: Task 782 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/ffa5ab2c-891f-4fc1-a396-f62629a6703f_calibrate_855911_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/ffa5ab2c-891f-4fc1-a396-f62629a6703f_calibrate_855911_24.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:20 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 877 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d1450f298434df98b4971908452b053e with result from task 877
parsl.dataflow.dflow DEBUG: Task 877 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 878 has memoization hash 1e1c8c307d94addedd3bcf839e213be2
parsl.dataflow.memoization INFO: Task 878 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 878 try 0 launched on executor multi with executor id 955
parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/e6d8b20c-6ffa-4c93-94e2-34edd4ec0a4f_calibrate_855909_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/e6d8b20c-6ffa-4c93-94e2-34edd4ec0a4f_calibrate_855909_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/9fad771b-2ef1-4846-b514-777fe8f70f6a_characterizeImage_855909_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/9fad771b-2ef1-4846-b514-777fe8f70f6a_characterizeImage_855909_30.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:21 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 988 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5057527f2327a4cca4c80f235d754193 with result from task 988
parsl.dataflow.dflow DEBUG: Task 988 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 989 has memoization hash 1875cd44bc2185332f33885207eff6f8
parsl.dataflow.memoization INFO: Task 989 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 989 try 0 launched on executor multi with executor id 956
parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7f6acb16-e3a3-451b-a9f4-cf835f798c84_calibrate_855820_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7f6acb16-e3a3-451b-a9f4-cf835f798c84_calibrate_855820_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/06755dea-f6ce-4dd0-a7f1-2923082f7246_characterizeImage_855820_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/06755dea-f6ce-4dd0-a7f1-2923082f7246_characterizeImage_855820_21.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 683 try 1 failed
parsl.dataflow.dflow ERROR: Task 683 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 5de2266b5b1128188d2d09a3c436bf7f with result from task 683
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/fc2d8795-2bbf-4d69-8144-d455c33bee2e_calibrate_855911_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/fc2d8795-2bbf-4d69-8144-d455c33bee2e_calibrate_855911_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:21 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 412 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6f0c07b36a62ab9476a579a819a49ccc with result from task 412
parsl.dataflow.dflow DEBUG: Task 412 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 413 has memoization hash 7036c557c6881346572f3b40d4c512b3
parsl.dataflow.memoization INFO: Task 413 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 413 try 0 launched on executor multi with executor id 957
parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/e306d769-6d1e-462a-9a5b-709e53c845a6_calibrate_855820_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/e306d769-6d1e-462a-9a5b-709e53c845a6_calibrate_855820_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/3871eafb-8810-4b9a-9f79-889af4c4e439_characterizeImage_855820_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/3871eafb-8810-4b9a-9f79-889af4c4e439_characterizeImage_855820_52.stderr
parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c471f458ad5e84d268039fe506c1f4a7 with result from task 266
parsl.dataflow.dflow DEBUG: Task 266 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/223ec596-d277-4fad-bd4f-a63435d42f37_calibrate_855909_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/223ec596-d277-4fad-bd4f-a63435d42f37_calibrate_855909_22.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:22 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 844 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 30199eda9364927796332c124d0d782e with result from task 844
parsl.dataflow.dflow DEBUG: Task 844 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 845 has memoization hash e13ff1cb41a04d80fd64cb75a862164b
parsl.dataflow.memoization INFO: Task 845 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15897a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 845 try 0 launched on executor multi with executor id 958
parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/546438f2-7efd-4dbd-aad3-9084b26c8416_calibrate_855909_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/546438f2-7efd-4dbd-aad3-9084b26c8416_calibrate_855909_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/1ffe8752-e66b-4e61-9531-ed3c1f03b719_characterizeImage_855909_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/1ffe8752-e66b-4e61-9531-ed3c1f03b719_characterizeImage_855909_23.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:23 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 772 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f3425e06005113529c06063cee997bf5 with result from task 772
parsl.dataflow.dflow DEBUG: Task 772 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 773 has memoization hash 585701f6714889fa6794a0ce5a3568dc
parsl.dataflow.memoization INFO: Task 773 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 773 try 0 launched on executor multi with executor id 959
parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/c196debe-15f4-4c1d-b488-a9378e1c231c_calibrate_855820_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/c196debe-15f4-4c1d-b488-a9378e1c231c_calibrate_855820_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/9ca65032-6e57-48cd-9c98-5a816016a451_characterizeImage_855820_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/9ca65032-6e57-48cd-9c98-5a816016a451_characterizeImage_855820_31.stderr
parsl.dataflow.dflow INFO: Task 1112 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 887f3a6730a2f11673fb0dfde7f74921 with result from task 1112
parsl.dataflow.dflow DEBUG: Task 1112 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/a1a12eb4-0a12-4d74-9026-e3c232666a28_calibrate_855910_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/a1a12eb4-0a12-4d74-9026-e3c232666a28_calibrate_855910_24.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 278 try 1 failed
parsl.dataflow.dflow ERROR: Task 278 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 77418da00e107e98d91f7e21e160d790 with result from task 278
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/f1f01e01-2819-48d4-97be-cb18f8a0396d_calibrate_855911_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/f1f01e01-2819-48d4-97be-cb18f8a0396d_calibrate_855911_34.stderr
parsl.dataflow.dflow INFO: Task 923 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 46584b3691fc2c2ce36ab1788d4e5afe with result from task 923
parsl.dataflow.dflow DEBUG: Task 923 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/ec8691f7-378b-44c4-9198-7f17773f0433_calibrate_855910_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/ec8691f7-378b-44c4-9198-7f17773f0433_calibrate_855910_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:23 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 856 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 50323c4d45677dfd476c835927b8c49f with result from task 856
parsl.dataflow.dflow DEBUG: Task 856 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 857 has memoization hash 08bb0d9cda6607c88dd36a412870fa72
parsl.dataflow.memoization INFO: Task 857 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 857 try 0 launched on executor multi with executor id 960
parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/78e63937-f7b6-4e11-9433-d785733cb762_calibrate_855820_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/78e63937-f7b6-4e11-9433-d785733cb762_calibrate_855820_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/7eb975e2-6856-4b04-9835-51b298f27403_characterizeImage_855820_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/7eb975e2-6856-4b04-9835-51b298f27403_characterizeImage_855820_35.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 1079 try 1 failed
parsl.dataflow.dflow ERROR: Task 1079 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 37125ccf9481580a8c8810817e24682a with result from task 1079
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/ebead29f-f80d-4a82-a148-4622ec13a647_calibrate_855911_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/ebead29f-f80d-4a82-a148-4622ec13a647_calibrate_855911_9.stderr
parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7fb7a8136d050578eea08b0a84c4111f with result from task 50
parsl.dataflow.dflow DEBUG: Task 50 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/9c673469-c840-4ea4-955a-2641cff95a23_calibrate_855909_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/9c673469-c840-4ea4-955a-2641cff95a23_calibrate_855909_39.stderr
parsl.dataflow.dflow INFO: Task 472 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8fddaa93043b689d22af16fb82a8deab with result from task 472
parsl.dataflow.dflow DEBUG: Task 472 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 473 has memoization hash 19085240caa75c4f67bde7e9df9f0385
parsl.dataflow.memoization INFO: Task 473 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 473 try 0 launched on executor multi with executor id 961
parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/ef148262-72ea-49b6-9065-d9fb1c1c32ed_calibrate_855820_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/ef148262-72ea-49b6-9065-d9fb1c1c32ed_calibrate_855820_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ebe408bc-8c11-4fc1-b73a-b70a37cd4cb3_characterizeImage_855820_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ebe408bc-8c11-4fc1-b73a-b70a37cd4cb3_characterizeImage_855820_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:24 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 1100 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 754f1c05731fe3bff5825022ece9eff3 with result from task 1100
parsl.dataflow.dflow DEBUG: Task 1100 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/eb1d3442-c51b-400e-9078-dba7823e7b6d_calibrate_855910_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/eb1d3442-c51b-400e-9078-dba7823e7b6d_calibrate_855910_18.stderr
parsl.dataflow.dflow INFO: Task 1013 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4f709b29442fac8ecc67cf513272547d with result from task 1013
parsl.dataflow.dflow DEBUG: Task 1013 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/82682231-7401-4da9-bc10-c75a514bb2b5_calibrate_855910_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/82682231-7401-4da9-bc10-c75a514bb2b5_calibrate_855910_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:25 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 881 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6d8eee6008bfb4639d4d36ca8da915ee with result from task 881
parsl.dataflow.dflow DEBUG: Task 881 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/e6a62c44-1893-4375-980f-338b51b63808_calibrate_855911_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/e6a62c44-1893-4375-980f-338b51b63808_calibrate_855911_36.stderr
parsl.dataflow.dflow INFO: Task 809 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a18109fc2eed485952c67e95892e3673 with result from task 809
parsl.dataflow.dflow DEBUG: Task 809 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/e712dba9-62ad-405d-8445-bb628a4fb410_calibrate_855910_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/e712dba9-62ad-405d-8445-bb628a4fb410_calibrate_855910_44.stderr
parsl.dataflow.dflow INFO: Task 524 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 417283f4039f481ae2d3d3d586b7cfc9 with result from task 524
parsl.dataflow.dflow DEBUG: Task 524 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/d664740a-86e0-4343-8188-4278e80bf8ad_calibrate_855909_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/d664740a-86e0-4343-8188-4278e80bf8ad_calibrate_855909_14.stderr
parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1c9cc27681a79947383e9036fb625d31 with result from task 305
parsl.dataflow.dflow DEBUG: Task 305 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/21e1bdd4-1597-4c3e-8ce9-a231a75a2da1_calibrate_855909_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/21e1bdd4-1597-4c3e-8ce9-a231a75a2da1_calibrate_855909_5.stderr
parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 68c2d5f4df76b57eba5575fe5e2edf5a with result from task 113
parsl.dataflow.dflow DEBUG: Task 113 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/857a3d8b-571e-4fba-9359-80fc9e1d4c7b_calibrate_855909_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/857a3d8b-571e-4fba-9359-80fc9e1d4c7b_calibrate_855909_51.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:25 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:26 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 956 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f5f83d77e25938e9a726e9693359c4f2 with result from task 956
parsl.dataflow.dflow DEBUG: Task 956 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/1b51178e-7f99-4931-8f2d-d68b3756c0ac_calibrate_855909_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/1b51178e-7f99-4931-8f2d-d68b3756c0ac_calibrate_855909_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:26 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 733 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 133780d1008f9ef703c15735db3a5fdc with result from task 733
parsl.dataflow.dflow DEBUG: Task 733 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 734 has memoization hash 757393eb81961e41bb661ca4d5c42c73
parsl.dataflow.memoization INFO: Task 734 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15858680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 734 try 0 launched on executor multi with executor id 962
parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/a62d0d07-e493-4824-ae7c-8295fe3b8776_calibrate_855820_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/a62d0d07-e493-4824-ae7c-8295fe3b8776_calibrate_855820_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ab459aa0-b32a-44ff-8287-fc6c65ff6421_characterizeImage_855820_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ab459aa0-b32a-44ff-8287-fc6c65ff6421_characterizeImage_855820_30.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:27 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:28 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 530 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e2e58240a1ce8f04c6aed629be26e8a2 with result from task 530
parsl.dataflow.dflow DEBUG: Task 530 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/1ab6df63-02b9-4b71-b934-33d54b68a440_calibrate_855910_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/1ab6df63-02b9-4b71-b934-33d54b68a440_calibrate_855910_36.stderr
parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3e512ad8a52b83510ec1ab4f2a8bd94c with result from task 121
parsl.dataflow.dflow DEBUG: Task 121 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 7cddc0cde5cdc63aa64309ff30ecbcbb
parsl.dataflow.memoization INFO: Task 122 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a40680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 963
parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/6197dd97-be2b-42f0-8c86-c79adbaf5799_calibrate_855820_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/6197dd97-be2b-42f0-8c86-c79adbaf5799_calibrate_855820_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/da5eed5a-7ad1-4ef6-9cc0-dd0c21c150bb_characterizeImage_855820_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/da5eed5a-7ad1-4ef6-9cc0-dd0c21c150bb_characterizeImage_855820_24.stderr
parsl.dataflow.dflow INFO: Task 1028 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d3fa3475141a567da8e08f2e4c5b3215 with result from task 1028
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:28 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow DEBUG: Task 1028 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/4bf3eda4-16a7-4f3a-8b23-72c6381ea8bc_calibrate_855909_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/4bf3eda4-16a7-4f3a-8b23-72c6381ea8bc_calibrate_855909_15.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:29 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 908 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0b0eb53b84123c3b53a3cef5bfe22f6e with result from task 908
parsl.dataflow.dflow DEBUG: Task 908 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/f7ca1d25-6cd1-4816-acea-0d442c2df65a_calibrate_855910_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/f7ca1d25-6cd1-4816-acea-0d442c2df65a_calibrate_855910_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:29 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 152 try 1 failed
parsl.dataflow.dflow ERROR: Task 152 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 79ef392d2714494703d5ab4beb319402 with result from task 152
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/137f21d7-a61d-4a26-953c-bf98c691df7e_calibrate_855911_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855911/137f21d7-a61d-4a26-953c-bf98c691df7e_calibrate_855911_41.stderr
parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 52a5fd5e05a1d3880f6e602738325298 with result from task 41
parsl.dataflow.dflow DEBUG: Task 41 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/92f33fd3-0c59-4962-b23a-a34e2bc231fa_calibrate_855909_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/92f33fd3-0c59-4962-b23a-a34e2bc231fa_calibrate_855909_31.stderr
parsl.dataflow.dflow INFO: Task 548 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d77328c43dd426a539c03826336d0b07 with result from task 548
parsl.dataflow.dflow DEBUG: Task 548 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/45144510-d84f-4061-bd54-9b99a8fdb531_calibrate_855910_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/45144510-d84f-4061-bd54-9b99a8fdb531_calibrate_855910_62.stderr
parsl.dataflow.dflow INFO: Task 884 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry db1ca80406d5b8228b6d65a91a6d2c1f with result from task 884
parsl.dataflow.dflow DEBUG: Task 884 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/404e9db8-6b7b-487e-aea5-930cb39408ff_calibrate_855910_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/404e9db8-6b7b-487e-aea5-930cb39408ff_calibrate_855910_38.stderr
parsl.dataflow.dflow INFO: Task 647 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7474f37e89b63bbfb6f47ac580398836 with result from task 647
parsl.dataflow.dflow DEBUG: Task 647 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/6ded6f27-f613-4785-8a8d-9a63d58105b1_calibrate_855909_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/6ded6f27-f613-4785-8a8d-9a63d58105b1_calibrate_855909_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:32 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:32 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a9e21cea46d38bf050616593983eef4b with result from task 128
parsl.dataflow.dflow DEBUG: Task 128 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/c491572c-097b-4878-9d6a-49b8ad3ed565_calibrate_855909_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/c491572c-097b-4878-9d6a-49b8ad3ed565_calibrate_855909_60.stderr
parsl.dataflow.dflow INFO: Task 704 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5983366c8a670ddfa1b8fc722ee4cc67 with result from task 704
parsl.dataflow.dflow DEBUG: Task 704 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/59fcef67-c828-4464-9924-dc389ff4092d_calibrate_855909_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/59fcef67-c828-4464-9924-dc389ff4092d_calibrate_855909_27.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:33 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 995 try 1 failed
parsl.dataflow.dflow ERROR: Task 995 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 795c8e9c014014f25edf91247f98867b with result from task 995
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/0edb750c-e078-4167-8edb-5314141fafdc_calibrate_855910_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/0edb750c-e078-4167-8edb-5314141fafdc_calibrate_855910_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:33 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1045 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1fb7e3d86cbe569b3408e48585cf6d33 with result from task 1045
parsl.dataflow.dflow DEBUG: Task 1045 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1046 has memoization hash 037349b1b8dc50f4b88049829999534b
parsl.dataflow.memoization INFO: Task 1046 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1046 try 0 launched on executor multi with executor id 964
parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/afe75ef8-740d-40fc-bfab-b3450957e5c8_calibrate_855820_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/afe75ef8-740d-40fc-bfab-b3450957e5c8_calibrate_855820_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/d3e6683a-abd7-429f-b11c-82eb23369091_characterizeImage_855820_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/d3e6683a-abd7-429f-b11c-82eb23369091_characterizeImage_855820_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:34 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 932 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d25546466483eb3d9532705bc1958759 with result from task 932
parsl.dataflow.dflow DEBUG: Task 932 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/fed42ed8-966a-47db-8904-197fe07539cf_calibrate_855909_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/fed42ed8-966a-47db-8904-197fe07539cf_calibrate_855909_32.stderr
parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e50a8d9251565a8be5792ba9cce50354 with result from task 70
parsl.dataflow.dflow DEBUG: Task 70 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 71 has memoization hash e5eb05936eb1ef2b3e5e712f321ec301
parsl.dataflow.memoization INFO: Task 71 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 965
parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/3e4d57fb-64d6-4e67-a6d5-6976a60fb13d_calibrate_855909_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/3e4d57fb-64d6-4e67-a6d5-6976a60fb13d_calibrate_855909_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/782dbd87-3469-4694-b922-6b1a5405e0ab_characterizeImage_855909_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/782dbd87-3469-4694-b922-6b1a5405e0ab_characterizeImage_855909_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:35 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:35 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:36 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0fb1bd52f29ec68d0ed4b0ac9208b78a with result from task 160
parsl.dataflow.dflow DEBUG: Task 160 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 1d4248a30b44b976b8ce524b79cf2630
parsl.dataflow.memoization INFO: Task 161 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a42f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 966
parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/918df869-b3d1-47af-bd35-ff9de151fa66_calibrate_855820_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/918df869-b3d1-47af-bd35-ff9de151fa66_calibrate_855820_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ad14a9c7-0779-40c7-9c26-b61b38e7d269_characterizeImage_855820_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ad14a9c7-0779-40c7-9c26-b61b38e7d269_characterizeImage_855820_1.stderr
parsl.dataflow.dflow INFO: Task 1025 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 403b590fc68b893db4c23e7770ed862c with result from task 1025
parsl.dataflow.dflow DEBUG: Task 1025 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/1326b4ae-86ab-4a14-b0b6-2c8eafe762e6_calibrate_855909_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/1326b4ae-86ab-4a14-b0b6-2c8eafe762e6_calibrate_855909_19.stderr
parsl.dataflow.dflow INFO: Task 488 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 91a9a8d8d7027aa9373391adf9267df8 with result from task 488
parsl.dataflow.dflow DEBUG: Task 488 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/15cbc0bc-dc9a-4d12-901d-6165f7619835_calibrate_855909_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/15cbc0bc-dc9a-4d12-901d-6165f7619835_calibrate_855909_16.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:36 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 777406fd0ff1ff8924b850de4258a8b7 with result from task 217
parsl.dataflow.dflow DEBUG: Task 217 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 218 has memoization hash d91711a897ed3d13d458eb984d243c67
parsl.dataflow.memoization INFO: Task 218 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a66ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 967
parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/c5362943-c05b-47f6-bf34-12e0a1fbb5aa_calibrate_855820_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/c5362943-c05b-47f6-bf34-12e0a1fbb5aa_calibrate_855820_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/d6b371d3-6370-45e8-94dd-becb9351d7c7_characterizeImage_855820_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/d6b371d3-6370-45e8-94dd-becb9351d7c7_characterizeImage_855820_13.stderr
parsl.dataflow.dflow INFO: Task 838 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4cf55512fe3a301645efca070f28c4b6 with result from task 838
parsl.dataflow.dflow DEBUG: Task 838 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 839 has memoization hash 87d93c955157c3f8a1fc37f8dbf00cfa
parsl.dataflow.memoization INFO: Task 839 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15897420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 839 try 0 launched on executor multi with executor id 968
parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/df762c07-8090-49b6-ac22-2299b88b5633_calibrate_855909_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/df762c07-8090-49b6-ac22-2299b88b5633_calibrate_855909_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/5b42ec12-14a4-41cd-937e-29587c8eb31b_characterizeImage_855909_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/5b42ec12-14a4-41cd-937e-29587c8eb31b_characterizeImage_855909_44.stderr
parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3accece77c75787f04ac55135ff1f658 with result from task 139
parsl.dataflow.dflow DEBUG: Task 139 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 8396ad2e99959dfa4bf126bf9c5daa6b
parsl.dataflow.memoization INFO: Task 140 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a41940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 969
parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/60de52e6-14bb-4755-a6f6-98f1d908801c_calibrate_855820_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/60de52e6-14bb-4755-a6f6-98f1d908801c_calibrate_855820_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/383918ae-d7ff-48a6-8b43-40d3b29fd103_characterizeImage_855820_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/383918ae-d7ff-48a6-8b43-40d3b29fd103_characterizeImage_855820_47.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:37 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 610 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9073f68d1ebf3d2a22109d22664fd6ce with result from task 610
parsl.dataflow.dflow DEBUG: Task 610 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 611 has memoization hash 0e1efb8fc40dc9ac9d24b908a515d67b
parsl.dataflow.memoization INFO: Task 611 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 611 try 0 launched on executor multi with executor id 970
parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/74c19f27-7242-4092-bc7a-61af040398eb_calibrate_855820_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/74c19f27-7242-4092-bc7a-61af040398eb_calibrate_855820_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/5750f29e-c021-4aaa-ae5b-09af6e2f5058_characterizeImage_855820_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/5750f29e-c021-4aaa-ae5b-09af6e2f5058_characterizeImage_855820_20.stderr
parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 31502b9e9f5a98344a992c4f6c4fff5d with result from task 23
parsl.dataflow.dflow DEBUG: Task 23 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/5342c5ab-2140-467b-92ab-15270e64fe2f_calibrate_855909_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/5342c5ab-2140-467b-92ab-15270e64fe2f_calibrate_855909_4.stderr
parsl.dataflow.dflow INFO: Task 602 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry efbf0c2e3e2a1c02637bd9803ef45f2d with result from task 602
parsl.dataflow.dflow DEBUG: Task 602 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/98c45b43-d52b-4bd3-a9fb-ab5f57e809e0_calibrate_855909_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/98c45b43-d52b-4bd3-a9fb-ab5f57e809e0_calibrate_855909_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:37 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 668 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 933a57a581c0e624ad59eaa84995e52c with result from task 668
parsl.dataflow.dflow DEBUG: Task 668 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/233ffee3-8888-4cca-8c2c-fabf05bb1e3a_calibrate_855909_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/233ffee3-8888-4cca-8c2c-fabf05bb1e3a_calibrate_855909_33.stderr
parsl.dataflow.dflow INFO: Task 976 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0c5c63b925090c1e642dbb652b95654e with result from task 976
parsl.dataflow.dflow DEBUG: Task 976 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 977 has memoization hash b8e95816646dba584d7a8655d670c5e6
parsl.dataflow.memoization INFO: Task 977 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 977 try 0 launched on executor multi with executor id 971
parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/caee272b-6a8c-422b-8c00-cf71813df75e_calibrate_855820_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/caee272b-6a8c-422b-8c00-cf71813df75e_calibrate_855820_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/2aac9e33-385b-4265-89fb-03d274996af0_characterizeImage_855820_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/2aac9e33-385b-4265-89fb-03d274996af0_characterizeImage_855820_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:38 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3a58909f8354fb231e893133c8766672 with result from task 158
parsl.dataflow.dflow DEBUG: Task 158 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/a1b3d2bb-1f45-4a5a-a340-7a95b245fde1_calibrate_855909_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/a1b3d2bb-1f45-4a5a-a340-7a95b245fde1_calibrate_855909_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:39 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0679a9f8ced9b8db612a67ddfd7cd052 with result from task 46
parsl.dataflow.dflow DEBUG: Task 46 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 0099714152e29b9f631a874cbe4662c3
parsl.dataflow.memoization INFO: Task 47 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169ff7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 972
parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/453b19ad-592b-445f-bb91-7441b49a4a45_calibrate_855820_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/453b19ad-592b-445f-bb91-7441b49a4a45_calibrate_855820_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/bbe7210e-b3fc-451c-9ae0-52ec4198fdeb_characterizeImage_855820_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/bbe7210e-b3fc-451c-9ae0-52ec4198fdeb_characterizeImage_855820_27.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:39 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:40 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1007 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3f1c8929d2df676f7e2aec2f67532067 with result from task 1007
parsl.dataflow.dflow DEBUG: Task 1007 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/ff2c9a80-a8b3-43a0-b429-2b2db4cd2384_calibrate_855909_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/ff2c9a80-a8b3-43a0-b429-2b2db4cd2384_calibrate_855909_21.stderr
parsl.dataflow.dflow INFO: Task 625 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d4831cbd6bff973fed130fcc1075df38 with result from task 625
parsl.dataflow.dflow DEBUG: Task 625 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 626 has memoization hash 2c555e2deef663235f3918ba0018fdf7
parsl.dataflow.memoization INFO: Task 626 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 626 try 0 launched on executor multi with executor id 973
parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/2b9f40ff-6a49-48ed-adcb-59bb495d3741_calibrate_855820_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/2b9f40ff-6a49-48ed-adcb-59bb495d3741_calibrate_855820_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/7e2f60b3-bac3-495e-8273-0545814d8b80_characterizeImage_855820_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/7e2f60b3-bac3-495e-8273-0545814d8b80_characterizeImage_855820_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:40 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:41 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 451 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5b05181bf527c2b92093634902ebdb63 with result from task 451
parsl.dataflow.dflow DEBUG: Task 451 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 452 has memoization hash 4b40b734eb8913f615ff6cb91be331e4
parsl.dataflow.memoization INFO: Task 452 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 452 try 0 launched on executor multi with executor id 974
parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/eff86f87-e2db-404b-bb28-3ceb24e4242d_calibrate_855820_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/eff86f87-e2db-404b-bb28-3ceb24e4242d_calibrate_855820_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/14ae06c2-d118-4820-9d9b-bde09099e8d9_characterizeImage_855820_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/14ae06c2-d118-4820-9d9b-bde09099e8d9_characterizeImage_855820_36.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:41 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a857bd2ed684b274cf118cdddfd71216 with result from task 212
parsl.dataflow.dflow DEBUG: Task 212 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/2d3dd28d-f2f1-44f1-a9ba-bd86be6169b1_calibrate_855909_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/2d3dd28d-f2f1-44f1-a9ba-bd86be6169b1_calibrate_855909_9.stderr
parsl.dataflow.dflow INFO: Task 965 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 870792b8bfb47a0e6fb22e1754846b0a with result from task 965
parsl.dataflow.dflow DEBUG: Task 965 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/14f395a7-a40f-4b36-b609-b1ec9acc5ed8_calibrate_855909_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/14f395a7-a40f-4b36-b609-b1ec9acc5ed8_calibrate_855909_40.stderr
parsl.dataflow.dflow INFO: Task 685 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 18da18ee359a20d44e24baec39273936 with result from task 685
parsl.dataflow.dflow DEBUG: Task 685 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 686 has memoization hash 949d221f3978246395ed7c263fdc4532
parsl.dataflow.memoization INFO: Task 686 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15809440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 686 try 0 launched on executor multi with executor id 975
parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/8fafc276-ba8a-4ea2-b48a-8f3c37b474f5_calibrate_855820_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/8fafc276-ba8a-4ea2-b48a-8f3c37b474f5_calibrate_855820_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c351ec92-f25c-4c33-ac33-be5c8974635e_characterizeImage_855820_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c351ec92-f25c-4c33-ac33-be5c8974635e_characterizeImage_855820_7.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:42 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1075 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 323fe88ad0c9f24da10de84133a61175 with result from task 1075
parsl.dataflow.dflow DEBUG: Task 1075 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1076 has memoization hash fb3104bd6b04ccda83cc8ccde48bbd22
parsl.dataflow.memoization INFO: Task 1076 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1076 try 0 launched on executor multi with executor id 976
parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/b8a91176-3047-49d9-9a10-af57fccfd54c_calibrate_855820_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/b8a91176-3047-49d9-9a10-af57fccfd54c_calibrate_855820_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/8213c3b3-9a1f-4fbd-a4ea-0d36d46029ce_characterizeImage_855820_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/8213c3b3-9a1f-4fbd-a4ea-0d36d46029ce_characterizeImage_855820_12.stderr
parsl.dataflow.dflow INFO: Task 592 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e7c425ee38fca5db753e4508dd5bc960 with result from task 592
parsl.dataflow.dflow DEBUG: Task 592 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 593 has memoization hash 657d460d21b4bde84c79e5bd64d1646a
parsl.dataflow.memoization INFO: Task 593 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 593 try 0 launched on executor multi with executor id 977
parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/194cd80d-2c83-465c-b668-fe790ad27c7b_calibrate_855820_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/194cd80d-2c83-465c-b668-fe790ad27c7b_calibrate_855820_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/b2445c1f-dd66-42b6-8aae-5ae3af029f68_characterizeImage_855820_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/b2445c1f-dd66-42b6-8aae-5ae3af029f68_characterizeImage_855820_9.stderr
parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6802c2c1b5ba6ff017cdf5ecfdc53507 with result from task 359
parsl.dataflow.dflow DEBUG: Task 359 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/f9906f03-8d59-4312-97a9-cbebab84a48a_calibrate_855909_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/f9906f03-8d59-4312-97a9-cbebab84a48a_calibrate_855909_24.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:42 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1090 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 49d52aa7747a92804d822c4d51818118 with result from task 1090
parsl.dataflow.dflow DEBUG: Task 1090 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1091 has memoization hash 79bb7f0591b21f498ee4362bde06bfeb
parsl.dataflow.memoization INFO: Task 1091 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1091 try 0 launched on executor multi with executor id 978
parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/3b1b5fc8-dc28-4e20-8876-0ce95cb849a2_calibrate_855820_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/3b1b5fc8-dc28-4e20-8876-0ce95cb849a2_calibrate_855820_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/95761f5c-07cf-4119-ba16-224e290484e3_characterizeImage_855820_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/95761f5c-07cf-4119-ba16-224e290484e3_characterizeImage_855820_14.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:43 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 851 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 35348700254419472a65f4c11bf3f6a5 with result from task 851
parsl.dataflow.dflow DEBUG: Task 851 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/5349dcb1-0784-41ce-a709-ef204397be47_calibrate_855909_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/5349dcb1-0784-41ce-a709-ef204397be47_calibrate_855909_13.stderr
parsl.dataflow.dflow INFO: Task 560 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f382ed284587e8d3ee313365fe01350c with result from task 560
parsl.dataflow.dflow DEBUG: Task 560 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/97c9bf25-35b5-4a56-8ae6-6cb62bf5672d_calibrate_855909_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/97c9bf25-35b5-4a56-8ae6-6cb62bf5672d_calibrate_855909_53.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:44 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:44 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 854 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a2e729d5ad88ae9d5f3c431c636138bb with result from task 854
parsl.dataflow.dflow DEBUG: Task 854 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/0019dc08-a14c-4fd8-b5c0-6448e19c1538_calibrate_855909_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/0019dc08-a14c-4fd8-b5c0-6448e19c1538_calibrate_855909_62.stderr
parsl.dataflow.dflow INFO: Task 583 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7dce9f9958dbc16b0b7bd88dc7d7483b with result from task 583
parsl.dataflow.dflow DEBUG: Task 583 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 584 has memoization hash 6640d994d2a709d08b484d7270f69047
parsl.dataflow.memoization INFO: Task 584 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 584 try 0 launched on executor multi with executor id 979
parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/e9e2fb6b-bef2-4aed-b67b-92f7c1e7b14d_calibrate_855820_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/e9e2fb6b-bef2-4aed-b67b-92f7c1e7b14d_calibrate_855820_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ffbbb12c-8d7c-48ea-a91f-14769608b6cc_characterizeImage_855820_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ffbbb12c-8d7c-48ea-a91f-14769608b6cc_characterizeImage_855820_48.stderr
parsl.dataflow.dflow INFO: Task 631 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cf412a4688e2207429da7d5252ffdd5c with result from task 631
parsl.dataflow.dflow DEBUG: Task 631 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 632 has memoization hash b9cbcccc5d89340f5ee218eec5686f93
parsl.dataflow.memoization INFO: Task 632 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 632 try 0 launched on executor multi with executor id 980
parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/e686b97d-f026-4bd6-aa8f-87d6f6a104f3_calibrate_855820_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/e686b97d-f026-4bd6-aa8f-87d6f6a104f3_calibrate_855820_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c2ddb1fe-ab98-466b-9ddd-90bee6f60a42_characterizeImage_855820_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c2ddb1fe-ab98-466b-9ddd-90bee6f60a42_characterizeImage_855820_8.stderr
parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c20312bafa97dd2821fd4753ce76b6c5 with result from task 131
parsl.dataflow.dflow DEBUG: Task 131 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/fb28739e-25bc-4102-9491-ebde27985bb4_calibrate_855909_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/fb28739e-25bc-4102-9491-ebde27985bb4_calibrate_855909_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:45 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8b30ee715ffa0e6b3a65d01a8c0532db with result from task 323
parsl.dataflow.dflow DEBUG: Task 323 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/0eee26b4-1b3b-4c12-9481-8c88c1fb2691_calibrate_855909_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/0eee26b4-1b3b-4c12-9481-8c88c1fb2691_calibrate_855909_17.stderr
parsl.dataflow.dflow INFO: Task 962 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 71c8bdf1676e35be1fcc53c14648bd1d with result from task 962
parsl.dataflow.dflow DEBUG: Task 962 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/2209d7fc-9fed-4c05-b1ab-6b1b0bd447d0_calibrate_855909_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/2209d7fc-9fed-4c05-b1ab-6b1b0bd447d0_calibrate_855909_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:45 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3128c04064b386f0d665b6b57ceebaff with result from task 38
parsl.dataflow.dflow DEBUG: Task 38 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/4c7d7f4a-f1be-495c-abf7-d1bc3e736166_calibrate_855909_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/4c7d7f4a-f1be-495c-abf7-d1bc3e736166_calibrate_855909_34.stderr
parsl.dataflow.dflow INFO: Task 466 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2de54ea0cfc93a9e192d91b7c5163f14 with result from task 466
parsl.dataflow.dflow DEBUG: Task 466 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 467 has memoization hash 584a36f9bc0d2d5deb3ccd4a2c00615b
parsl.dataflow.memoization INFO: Task 467 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 467 try 0 launched on executor multi with executor id 981
parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/a24e0007-a778-40e0-ad80-98a39d476a1a_calibrate_855820_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/a24e0007-a778-40e0-ad80-98a39d476a1a_calibrate_855820_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/032cca53-7839-4440-88cc-6efbdb62717c_characterizeImage_855820_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/032cca53-7839-4440-88cc-6efbdb62717c_characterizeImage_855820_16.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:46 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1040 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0e789652a1c39b090b34aaa0c44dbc64 with result from task 1040
parsl.dataflow.dflow DEBUG: Task 1040 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/f408d0d8-18b1-4eaa-9626-c36503685bc3_calibrate_855820_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/f408d0d8-18b1-4eaa-9626-c36503685bc3_calibrate_855820_39.stderr
parsl.dataflow.dflow INFO: Task 1016 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2a499d40ead93cc26fab893e6b404a77 with result from task 1016
parsl.dataflow.dflow DEBUG: Task 1016 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/2cb92a27-c5da-4f07-b252-0b6082130fbf_calibrate_855909_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/2cb92a27-c5da-4f07-b252-0b6082130fbf_calibrate_855909_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:46 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:47 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 889 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 51517bde30808a0da7ea8842fb638568 with result from task 889
parsl.dataflow.dflow DEBUG: Task 889 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 890 has memoization hash 8e4e95439ea526c5108fcfab58c4b3af
parsl.dataflow.memoization INFO: Task 890 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156ea980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 890 try 0 launched on executor multi with executor id 982
parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/13bc01c6-7c4e-4cd8-81d9-2b1dd54475b4_calibrate_855820_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/13bc01c6-7c4e-4cd8-81d9-2b1dd54475b4_calibrate_855820_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/9aacd4d8-3585-4524-be65-b137cdda2dbb_characterizeImage_855820_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/9aacd4d8-3585-4524-be65-b137cdda2dbb_characterizeImage_855820_3.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:48 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:48 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:49 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:49 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 205
parsl.dataflow.strategy DEBUG: Executor multi has 205 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 20:57:50 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
parsl.dataflow.dflow INFO: Task 895 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 361af696bb66b227cee1094fb63d27e0 with result from task 895
parsl.dataflow.dflow DEBUG: Task 895 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 896 has memoization hash a62bd0367c8a32945d5e43ef9bb667be
parsl.dataflow.memoization INFO: Task 896 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156eafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 896 try 0 launched on executor multi with executor id 983
parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/8a36815c-2b22-484a-be85-6a72b3a4b74b_calibrate_855820_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/8a36815c-2b22-484a-be85-6a72b3a4b74b_calibrate_855820_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/b7760292-e48e-4788-95aa-882dd0522c01_characterizeImage_855820_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/b7760292-e48e-4788-95aa-882dd0522c01_characterizeImage_855820_60.stderr
2024-03-09 20:57:50 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 659 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b6800f01a228e5b9095bc20516e3a3b6 with result from task 659
parsl.dataflow.dflow DEBUG: Task 659 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/e2df8168-7039-4116-a590-a16d57a43e19_calibrate_855909_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/e2df8168-7039-4116-a590-a16d57a43e19_calibrate_855909_52.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 866 try 1 failed
parsl.dataflow.dflow ERROR: Task 866 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry b064d4ad890c1f762a3b34fa1ddc30ba with result from task 866
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/c747a1b1-fdc0-44b6-b17d-9eced577e4c3_calibrate_855910_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855910/c747a1b1-fdc0-44b6-b17d-9eced577e4c3_calibrate_855910_35.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:50 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:51 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7ac38e7d30256d1c7605bfe53f6e8eb9 with result from task 43
parsl.dataflow.dflow DEBUG: Task 43 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 0534c268d95f66ede5a69b94273dd351
parsl.dataflow.memoization INFO: Task 44 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fe0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 984
parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/c48849aa-8a02-4d8e-bf1c-4fb90c62e4e6_calibrate_855820_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/c48849aa-8a02-4d8e-bf1c-4fb90c62e4e6_calibrate_855820_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/7c0cdb32-7c95-4250-a744-924d7e612461_characterizeImage_855820_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/7c0cdb32-7c95-4250-a744-924d7e612461_characterizeImage_855820_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:52 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ea5d22a6f4dbcf365da3493aa1f54f04 with result from task 214
parsl.dataflow.dflow DEBUG: Task 214 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 215 has memoization hash 5ab1c7508c6beea68d0ebc50b79140ce
parsl.dataflow.memoization INFO: Task 215 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a667a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 985
parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/cf674d0e-1649-46a2-8543-1fc4795fbad0_calibrate_855820_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/cf674d0e-1649-46a2-8543-1fc4795fbad0_calibrate_855820_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/0dc84a3a-9b16-4880-86a7-cbd6b07b9bf0_characterizeImage_855820_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/0dc84a3a-9b16-4880-86a7-cbd6b07b9bf0_characterizeImage_855820_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:52 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 826 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 11701675ab424a79fe16d8818f374d21 with result from task 826
parsl.dataflow.dflow DEBUG: Task 826 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 827 has memoization hash 7707f9c1728595bb088de4aeb02d17b4
parsl.dataflow.memoization INFO: Task 827 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158967a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 827 try 0 launched on executor multi with executor id 986
parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/1f22c968-ab71-49cb-a122-71242b0a8b07_calibrate_855820_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/1f22c968-ab71-49cb-a122-71242b0a8b07_calibrate_855820_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/5e4fdc11-62ac-42b8-b4e4-51150a991cef_characterizeImage_855820_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/5e4fdc11-62ac-42b8-b4e4-51150a991cef_characterizeImage_855820_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:53 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 540122b77642909a697289944dd95cf3 with result from task 82
parsl.dataflow.dflow DEBUG: Task 82 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 30321e8cbaca630e68ac26ad710a397c
parsl.dataflow.memoization INFO: Task 83 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 987
parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/eac7ad90-585d-4b21-9141-353e09984a41_calibrate_855820_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/eac7ad90-585d-4b21-9141-353e09984a41_calibrate_855820_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/1414d790-0808-4fb3-a913-436ff16a524d_characterizeImage_855820_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/1414d790-0808-4fb3-a913-436ff16a524d_characterizeImage_855820_11.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:53 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:54 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 1093 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 984e331893d7d3f515bb828385c67c7b with result from task 1093
parsl.dataflow.dflow DEBUG: Task 1093 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1094 has memoization hash 5e92f0c0b144917fc49533ad9edae2de
parsl.dataflow.memoization INFO: Task 1094 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1094 try 0 launched on executor multi with executor id 988
parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/4d8d543f-3cb4-4bde-8311-ebe18d9ee388_calibrate_855820_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/4d8d543f-3cb4-4bde-8311-ebe18d9ee388_calibrate_855820_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/f683499d-05c9-46f7-8b35-32c3c36685d5_characterizeImage_855820_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/f683499d-05c9-46f7-8b35-32c3c36685d5_characterizeImage_855820_4.stderr
parsl.dataflow.dflow INFO: Task 860 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1fe529a12dc22844074b738714fcd222 with result from task 860
parsl.dataflow.dflow DEBUG: Task 860 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/77bc71e5-9fde-453e-8292-03e6e4f6cc5e_calibrate_855909_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/77bc71e5-9fde-453e-8292-03e6e4f6cc5e_calibrate_855909_35.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:54 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:55 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6ce10c567ea0550f2bb0f35ffe6493dc with result from task 263
parsl.dataflow.dflow DEBUG: Task 263 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/b05e4d11-5f49-421b-968a-e27937f7ee1e_calibrate_855820_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/b05e4d11-5f49-421b-968a-e27937f7ee1e_calibrate_855820_25.stderr
parsl.dataflow.dflow INFO: Task 887 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e5409d79154942472868d3ae3e80f1a9 with result from task 887
parsl.dataflow.dflow DEBUG: Task 887 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/dc9ff95b-7ae9-43f8-b845-187b6d1cfa13_calibrate_855909_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/dc9ff95b-7ae9-43f8-b845-187b6d1cfa13_calibrate_855909_1.stderr
parsl.dataflow.dflow INFO: Task 817 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d030e67f2a06573c35fce35c559b1928 with result from task 817
parsl.dataflow.dflow DEBUG: Task 817 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 818 has memoization hash 33c216aa075fa4492a788b27a7c95a7f
parsl.dataflow.memoization INFO: Task 818 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15895e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 818 try 0 launched on executor multi with executor id 989
parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/929b43f0-347e-402c-bede-0fb884df93b6_calibrate_855820_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/929b43f0-347e-402c-bede-0fb884df93b6_calibrate_855820_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/f2c912ed-3c21-466e-8fe8-2479c91e6172_characterizeImage_855820_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/f2c912ed-3c21-466e-8fe8-2479c91e6172_characterizeImage_855820_40.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:56 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:56 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6ba5354201a8081028e025b78ae0d622 with result from task 368
parsl.dataflow.dflow DEBUG: Task 368 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/5eb16997-8be0-454f-9afe-0051a5dd918f_calibrate_855909_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/5eb16997-8be0-454f-9afe-0051a5dd918f_calibrate_855909_38.stderr
parsl.dataflow.dflow INFO: Task 554 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0df99efcb55612b6b2ff420a45071ea4 with result from task 554
parsl.dataflow.dflow DEBUG: Task 554 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/a1441880-e357-413c-a1c3-831e0383bbb0_calibrate_855909_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/a1441880-e357-413c-a1c3-831e0383bbb0_calibrate_855909_48.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:57 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e252698d54ee76437c0fb16b5cfa14fc with result from task 338
parsl.dataflow.dflow DEBUG: Task 338 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7388943f-22af-4226-b461-839a5b39d052_calibrate_855820_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7388943f-22af-4226-b461-839a5b39d052_calibrate_855820_10.stderr
parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 317c8cf672a9576dac277963d932a0ae with result from task 107
parsl.dataflow.dflow DEBUG: Task 107 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/3156187d-b017-45f9-ab8b-a9056fed69d5_calibrate_855909_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/3156187d-b017-45f9-ab8b-a9056fed69d5_calibrate_855909_49.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:57 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:58 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 1081 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b8b4a7b0708faafc466c57c8f744bf91 with result from task 1081
parsl.dataflow.dflow DEBUG: Task 1081 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1082 has memoization hash d24f4ee32601bb1818dc2f0cf6b61953
parsl.dataflow.memoization INFO: Task 1082 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1082 try 0 launched on executor multi with executor id 990
parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7a3ed83c-d0be-47c9-b5ef-04b3881baae4_calibrate_855820_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7a3ed83c-d0be-47c9-b5ef-04b3881baae4_calibrate_855820_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/5f330de0-4063-4425-a305-c24b04ae3132_characterizeImage_855820_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/5f330de0-4063-4425-a305-c24b04ae3132_characterizeImage_855820_49.stderr
parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1ab096999c2f3034f03dff188b245c42 with result from task 257
parsl.dataflow.dflow DEBUG: Task 257 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/4b7a2b56-926a-4719-8e17-77ff8d3bc299_calibrate_855820_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/4b7a2b56-926a-4719-8e17-77ff8d3bc299_calibrate_855820_33.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:58 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c8877a61dcaad6f463823d6e9ea9c84d with result from task 11
parsl.dataflow.dflow DEBUG: Task 11 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/e59b8adf-cc32-4181-bade-8b5a921f4fcf_calibrate_855820_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/e59b8adf-cc32-4181-bade-8b5a921f4fcf_calibrate_855820_32.stderr
parsl.dataflow.dflow INFO: Task 635 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 24ac138105e2f1f132aaea48c2376faf with result from task 635
parsl.dataflow.dflow DEBUG: Task 635 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/9ca73e40-4427-4446-87ba-7850aee51b63_calibrate_855820_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/9ca73e40-4427-4446-87ba-7850aee51b63_calibrate_855820_19.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:59 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:57:59 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 478 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a23686468a4c2dcf39dfa301f924b081 with result from task 478
parsl.dataflow.dflow DEBUG: Task 478 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 479 has memoization hash 5f0bb4892740ce3973a70afe549fefb9
parsl.dataflow.memoization INFO: Task 479 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 479 try 0 launched on executor multi with executor id 991
parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/5f60bd31-69ac-4864-b7a4-f8c12fce3f2c_calibrate_855820_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/5f60bd31-69ac-4864-b7a4-f8c12fce3f2c_calibrate_855820_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/787a3392-1133-43a9-aea9-8dc23a153ff9_characterizeImage_855820_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/787a3392-1133-43a9-aea9-8dc23a153ff9_characterizeImage_855820_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:00 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:00 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 1096 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b7ba27bc69650476e62eb444e9b7a9f9 with result from task 1096
parsl.dataflow.dflow DEBUG: Task 1096 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1097 has memoization hash 6bf2185c3a5dec040f65acdf8ee941f7
parsl.dataflow.memoization INFO: Task 1097 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1560c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1097 try 0 launched on executor multi with executor id 992
parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/5ed09bcc-77f1-492d-908b-93a5ab0ff576_calibrate_855820_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/5ed09bcc-77f1-492d-908b-93a5ab0ff576_calibrate_855820_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c6f9712a-9e9c-4a22-981f-a183a4c8dabe_characterizeImage_855820_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c6f9712a-9e9c-4a22-981f-a183a4c8dabe_characterizeImage_855820_22.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:01 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:02 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 779 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8d910dc61a407e13f6c648bbeffc6a79 with result from task 779
parsl.dataflow.dflow DEBUG: Task 779 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/e17dbe3c-57be-46bf-8b84-2ad2f19b6751_calibrate_855909_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/e17dbe3c-57be-46bf-8b84-2ad2f19b6751_calibrate_855909_8.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:02 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 502 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c1e40febcda5f859ac9b8ff6978d7a24 with result from task 502
parsl.dataflow.dflow DEBUG: Task 502 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 503 has memoization hash edc8df9403a05205124532335e949122
parsl.dataflow.memoization INFO: Task 503 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159454e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 503 try 0 launched on executor multi with executor id 993
parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/2453f2c0-3d23-4630-9577-6735ef9f3bad_calibrate_855820_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/2453f2c0-3d23-4630-9577-6735ef9f3bad_calibrate_855820_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c22bf194-67bd-4de3-b364-7fce9115fda1_characterizeImage_855820_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c22bf194-67bd-4de3-b364-7fce9115fda1_characterizeImage_855820_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:03 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 398 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 74efd003d93f9cae2d1ea4764acdbea2 with result from task 398
parsl.dataflow.dflow DEBUG: Task 398 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/5fe998ed-146a-4ca7-a5c5-429321432ad4_calibrate_855820_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/5fe998ed-146a-4ca7-a5c5-429321432ad4_calibrate_855820_45.stderr
parsl.dataflow.dflow INFO: Task 517 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ede42ae69dafb537e02179e558af1275 with result from task 517
parsl.dataflow.dflow DEBUG: Task 517 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 518 has memoization hash 3866a4b19335e7bfb80e44f68d686fc1
parsl.dataflow.memoization INFO: Task 518 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15946480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 518 try 0 launched on executor multi with executor id 994
parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/4de6f49d-c2f6-40ee-8ee3-1e2d880cb1d8_calibrate_855820_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/4de6f49d-c2f6-40ee-8ee3-1e2d880cb1d8_calibrate_855820_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/cd8000da-0d9d-4870-9e74-39cb1774ecf2_characterizeImage_855820_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/cd8000da-0d9d-4870-9e74-39cb1774ecf2_characterizeImage_855820_38.stderr
parsl.dataflow.dflow INFO: Task 419 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b44cdd0282b06ea0866a86becddaa0a1 with result from task 419
parsl.dataflow.dflow DEBUG: Task 419 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/a4aad4bb-ea20-43b8-b79d-3fcce866a3db_calibrate_855909_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/a4aad4bb-ea20-43b8-b79d-3fcce866a3db_calibrate_855909_41.stderr
parsl.dataflow.dflow INFO: Task 449 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 67b2e718b2e8332a7ba554bc96bad68f with result from task 449
parsl.dataflow.dflow DEBUG: Task 449 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/53649b60-be4c-4ec0-91b9-2e604bcd1a6c_calibrate_855909_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/53649b60-be4c-4ec0-91b9-2e604bcd1a6c_calibrate_855909_47.stderr
parsl.dataflow.dflow INFO: Task 1051 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f4f4ae53e691e288193be263e0160b5e with result from task 1051
parsl.dataflow.dflow DEBUG: Task 1051 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1052 has memoization hash 1b1d4e047b90f57bce30e4384c1b5aee
parsl.dataflow.memoization INFO: Task 1052 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1052 try 0 launched on executor multi with executor id 995
parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/f2f7ca35-31ab-42c8-b5ec-670c2dd106ff_calibrate_855820_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/f2f7ca35-31ab-42c8-b5ec-670c2dd106ff_calibrate_855820_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/2ac33e9b-d573-4950-96fe-3c84aca88ad2_characterizeImage_855820_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/2ac33e9b-d573-4950-96fe-3c84aca88ad2_characterizeImage_855820_53.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:05 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 706 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5be39bcd95a42545a73cfd57ae88775a with result from task 706
parsl.dataflow.dflow DEBUG: Task 706 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 707 has memoization hash 0741bb203a046cda5791247396622561
parsl.dataflow.memoization INFO: Task 707 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 707 try 0 launched on executor multi with executor id 996
parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/b9eed9a4-44b7-4164-b7b2-fff9a4ec1f61_calibrate_855820_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/b9eed9a4-44b7-4164-b7b2-fff9a4ec1f61_calibrate_855820_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c828e551-6af1-4f5a-bef9-12d0f23ab841_characterizeImage_855820_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c828e551-6af1-4f5a-bef9-12d0f23ab841_characterizeImage_855820_23.stderr
parsl.dataflow.dflow INFO: Task 910 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f21c3abfdaf034cda0458bb7fd4105d0 with result from task 910
parsl.dataflow.dflow DEBUG: Task 910 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 911 has memoization hash 306ecadc4f355aedd25912bec5488989
parsl.dataflow.memoization INFO: Task 911 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156ebec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 911 try 0 launched on executor multi with executor id 997
parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7e7fa924-166d-4f1d-aa0f-cc678bbcce90_calibrate_855820_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7e7fa924-166d-4f1d-aa0f-cc678bbcce90_calibrate_855820_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/9373a00d-22d0-4fc1-88f7-c296df4c2e2b_characterizeImage_855820_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/9373a00d-22d0-4fc1-88f7-c296df4c2e2b_characterizeImage_855820_34.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:06 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 482 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry db6992591095a1e6ae94fd9fc7b6a691 with result from task 482
parsl.dataflow.dflow DEBUG: Task 482 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/e47f955e-899e-483e-82f0-d459c497afc8_calibrate_855909_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/e47f955e-899e-483e-82f0-d459c497afc8_calibrate_855909_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:06 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 718 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f077e85d55c4299c61411034090a781c with result from task 718
parsl.dataflow.dflow DEBUG: Task 718 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 719 has memoization hash 19040efc82b8f1f532306f6f4541c618
parsl.dataflow.memoization INFO: Task 719 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 719 try 0 launched on executor multi with executor id 998
parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/61bb2b01-9aa7-4e4b-8205-ec8cd15de682_calibrate_855820_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/61bb2b01-9aa7-4e4b-8205-ec8cd15de682_calibrate_855820_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ea0cf7a9-f65a-4fd2-b5fe-88cc7369b661_characterizeImage_855820_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/ea0cf7a9-f65a-4fd2-b5fe-88cc7369b661_characterizeImage_855820_41.stderr
parsl.dataflow.dflow INFO: Task 800 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 56b7ffed9adf07c82f988fcecb394772 with result from task 800
parsl.dataflow.dflow DEBUG: Task 800 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/49fe7c1e-b023-4d43-aa87-2f716d5a7b39_calibrate_855909_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/49fe7c1e-b023-4d43-aa87-2f716d5a7b39_calibrate_855909_3.stderr
parsl.dataflow.dflow INFO: Task 836 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a74b33752a9e5f4265e2b2edd12924fc with result from task 836
parsl.dataflow.dflow DEBUG: Task 836 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/169b8ff4-c9f5-467f-bccc-39140eadfe93_calibrate_855909_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/169b8ff4-c9f5-467f-bccc-39140eadfe93_calibrate_855909_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:07 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 497 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 79417711c86f8c8dd2fc59c8e905fb4a with result from task 497
parsl.dataflow.dflow DEBUG: Task 497 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/6f6e6793-4b80-4621-9311-c67badbdcdf3_calibrate_855909_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/6f6e6793-4b80-4621-9311-c67badbdcdf3_calibrate_855909_18.stderr
parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d7460398fd4d5ffe05578a84a887b15b with result from task 170
parsl.dataflow.dflow DEBUG: Task 170 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/afa4aaff-f88b-40d2-b750-e8a690ffe052_calibrate_855909_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/afa4aaff-f88b-40d2-b750-e8a690ffe052_calibrate_855909_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:07 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:08 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c72d3eb61df749b74d5376420f48a018 with result from task 190
parsl.dataflow.dflow DEBUG: Task 190 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 191 has memoization hash ed79657f3c87999e4aa3f54e82442918
parsl.dataflow.memoization INFO: Task 191 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a64ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 999
parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/b9becbda-b490-49d5-a68c-1c58f5b390a7_calibrate_855820_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/b9becbda-b490-49d5-a68c-1c58f5b390a7_calibrate_855820_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/1c2b6108-4f77-4ad5-8b16-eb1d8b97c7d5_characterizeImage_855820_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/1c2b6108-4f77-4ad5-8b16-eb1d8b97c7d5_characterizeImage_855820_29.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:08 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:09 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:09 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.dataflow.dflow INFO: Task 746 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2a8a40f5b5b07ef99dbf7ba1d80f69d7 with result from task 746
parsl.dataflow.dflow DEBUG: Task 746 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/7419e133-b980-4e4e-82de-f7509b677834_calibrate_855909_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/7419e133-b980-4e4e-82de-f7509b677834_calibrate_855909_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:10 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:10 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.dataflow.dflow INFO: Task 845 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e13ff1cb41a04d80fd64cb75a862164b with result from task 845
parsl.dataflow.dflow DEBUG: Task 845 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/546438f2-7efd-4dbd-aad3-9084b26c8416_calibrate_855909_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/546438f2-7efd-4dbd-aad3-9084b26c8416_calibrate_855909_23.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:11 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:11 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:12 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:12 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:13 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:13 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:14 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 587 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 16029cf2744ca4dc98c647dc0128e79a with result from task 587
parsl.dataflow.dflow DEBUG: Task 587 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/6d4625b2-b68f-4f17-830e-f699211f7281_calibrate_855909_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/6d4625b2-b68f-4f17-830e-f699211f7281_calibrate_855909_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:14 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7adcf9beece119f1af2bf82d13afc106 with result from task 164
parsl.dataflow.dflow DEBUG: Task 164 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/34155c04-de53-4a83-b402-959b4958ddf7_calibrate_855820_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/34155c04-de53-4a83-b402-959b4958ddf7_calibrate_855820_26.stderr
parsl.dataflow.dflow INFO: Task 620 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 504c2bd0f04f626b71731df74e3a1f7e with result from task 620
parsl.dataflow.dflow DEBUG: Task 620 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/33190932-3f3c-407f-8c9d-17f84c38b814_calibrate_855909_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/33190932-3f3c-407f-8c9d-17f84c38b814_calibrate_855909_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:15 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:15 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:15 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:16 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:16 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ec3fb2162981d5b858eb24e266ebcc01 with result from task 88
parsl.dataflow.dflow DEBUG: Task 88 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 89 has memoization hash cd18a9f1623a030175e9dd5a868557f2
parsl.dataflow.memoization INFO: Task 89 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 1000
parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/71a060e3-794c-4c0b-be11-e68a75bb42d6_calibrate_855820_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/71a060e3-794c-4c0b-be11-e68a75bb42d6_calibrate_855820_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/5daa879c-b376-49e2-a408-eebd8244eeeb_characterizeImage_855820_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/5daa879c-b376-49e2-a408-eebd8244eeeb_characterizeImage_855820_43.stderr
parsl.dataflow.dflow INFO: Task 484 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f7db7810fdee511978569b5c27c71b33 with result from task 484
parsl.dataflow.dflow DEBUG: Task 484 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 485 has memoization hash 4c7d5d59b3994fdf4546f0e10178cd53
parsl.dataflow.memoization INFO: Task 485 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15944220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 485 try 0 launched on executor multi with executor id 1001
parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/a12339cc-ea8e-4314-a1fb-95024fb406d8_calibrate_855820_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/a12339cc-ea8e-4314-a1fb-95024fb406d8_calibrate_855820_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/895d27e2-90dc-4a78-80cc-6f08841898e2_characterizeImage_855820_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/895d27e2-90dc-4a78-80cc-6f08841898e2_characterizeImage_855820_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:16 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 661 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 383d3883492a463a23a066bcfaf3b99e with result from task 661
parsl.dataflow.dflow DEBUG: Task 661 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 662 has memoization hash e8fe1226f6dec28c93bb64cca8483c0f
parsl.dataflow.memoization INFO: Task 662 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 662 try 0 launched on executor multi with executor id 1002
parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/9df68c8d-9727-4ba4-95c0-4bfe41ae74df_calibrate_855820_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/9df68c8d-9727-4ba4-95c0-4bfe41ae74df_calibrate_855820_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/aa3cef68-3e27-434f-97e3-def31bd73121_characterizeImage_855820_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/aa3cef68-3e27-434f-97e3-def31bd73121_characterizeImage_855820_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:16 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:17 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:17 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:17 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:17 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:18 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:18 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1036 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c99569d9b742a0c986ea77c05574387f with result from task 1036
parsl.dataflow.dflow DEBUG: Task 1036 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1037 has memoization hash 0592e4ee549daebc4794faa224e41ff5
parsl.dataflow.memoization INFO: Task 1037 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1037 try 0 launched on executor multi with executor id 1003
parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/ae1338cd-97e8-4620-aac3-428cfd99f532_calibrate_855820_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/ae1338cd-97e8-4620-aac3-428cfd99f532_calibrate_855820_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/f2069b30-9019-4363-9b93-edf5c3ec0866_characterizeImage_855820_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/f2069b30-9019-4363-9b93-edf5c3ec0866_characterizeImage_855820_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:18 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:18 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:19 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:19 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:19 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:19 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:20 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:20 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:20 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:21 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:21 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:21 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:21 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:22 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 773 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 585701f6714889fa6794a0ce5a3568dc with result from task 773
parsl.dataflow.dflow DEBUG: Task 773 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/c196debe-15f4-4c1d-b488-a9378e1c231c_calibrate_855820_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/c196debe-15f4-4c1d-b488-a9378e1c231c_calibrate_855820_31.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:22 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 569 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9e1dcee26d7e769f187bc7e461b7a3d6 with result from task 569
parsl.dataflow.dflow DEBUG: Task 569 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7f50291a-a786-4033-a024-b5ab0f53255b_calibrate_855820_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7f50291a-a786-4033-a024-b5ab0f53255b_calibrate_855820_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:22 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:22 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:23 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:23 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:23 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:23 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:24 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:24 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:24 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:25 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:25 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:25 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:25 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2b64f44f87a74058a58e3de9a0a7b218 with result from task 172
parsl.dataflow.dflow DEBUG: Task 172 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 173 has memoization hash dedec706343a0484aa2790a9b39b7a92
parsl.dataflow.memoization INFO: Task 173 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a43ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 1004
parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/c62d82d3-c55f-46fd-948f-c5e304d9bb89_calibrate_855820_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/c62d82d3-c55f-46fd-948f-c5e304d9bb89_calibrate_855820_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/49cb6f63-543d-4e57-a501-5cd8739737fa_characterizeImage_855820_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/49cb6f63-543d-4e57-a501-5cd8739737fa_characterizeImage_855820_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:26 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:26 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 989 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1875cd44bc2185332f33885207eff6f8 with result from task 989
parsl.dataflow.dflow DEBUG: Task 989 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7f6acb16-e3a3-451b-a9f4-cf835f798c84_calibrate_855820_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7f6acb16-e3a3-451b-a9f4-cf835f798c84_calibrate_855820_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:26 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:26 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:27 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:27 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 796 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0595e7baa35dcec787b5608967f1b535 with result from task 796
parsl.dataflow.dflow DEBUG: Task 796 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 797 has memoization hash 7850d53773bedaf7197c12a469a861f9
parsl.dataflow.memoization INFO: Task 797 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15894860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 797 try 0 launched on executor multi with executor id 1005
parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/2f79bc85-e74b-4bf8-937d-1aa18b746d7e_calibrate_855820_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/2f79bc85-e74b-4bf8-937d-1aa18b746d7e_calibrate_855820_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c3160f01-f18d-4235-be9b-9e9d350031b0_characterizeImage_855820_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/c3160f01-f18d-4235-be9b-9e9d350031b0_characterizeImage_855820_50.stderr
parsl.dataflow.dflow INFO: Task 1103 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4edf0279be1c8d675d1a3dbfe0050aa5 with result from task 1103
parsl.dataflow.dflow DEBUG: Task 1103 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/317684b9-714b-48e6-941d-3b739e500022_calibrate_855820_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/317684b9-714b-48e6-941d-3b739e500022_calibrate_855820_51.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:27 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:27 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 175
parsl.dataflow.strategy DEBUG: Executor multi has 175 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 20:58:28 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 20:58:28 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:28 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:28 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 473 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 19085240caa75c4f67bde7e9df9f0385 with result from task 473
parsl.dataflow.dflow DEBUG: Task 473 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/ef148262-72ea-49b6-9065-d9fb1c1c32ed_calibrate_855820_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/ef148262-72ea-49b6-9065-d9fb1c1c32ed_calibrate_855820_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:29 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1015935011ba1aec5340da4f0e1cac67 with result from task 118
parsl.dataflow.dflow DEBUG: Task 118 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 1a1f297d54ec23a54877181af69ffd67
parsl.dataflow.memoization INFO: Task 119 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a40360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 1006
parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/0da864ba-e1ff-4ff6-88fe-7563d4db6311_calibrate_855822_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/0da864ba-e1ff-4ff6-88fe-7563d4db6311_calibrate_855822_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/c8cbb082-36a5-4cb4-a3fe-e6634f7ebe66_characterizeImage_855822_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/c8cbb082-36a5-4cb4-a3fe-e6634f7ebe66_characterizeImage_855822_19.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:29 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:29 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:29 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 461 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 664a4c4f975c8815a0eb87bae9c7451b with result from task 461
parsl.dataflow.dflow DEBUG: Task 461 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/068475ad-76f2-44f1-99b6-c862f396f8e9_calibrate_855820_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/068475ad-76f2-44f1-99b6-c862f396f8e9_calibrate_855820_15.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:30 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:30 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:30 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:30 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:31 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:31 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:31 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:32 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:32 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 839 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 87d93c955157c3f8a1fc37f8dbf00cfa with result from task 839
parsl.dataflow.dflow DEBUG: Task 839 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/df762c07-8090-49b6-ac22-2299b88b5633_calibrate_855909_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/df762c07-8090-49b6-ac22-2299b88b5633_calibrate_855909_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:32 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:32 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 878 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1e1c8c307d94addedd3bcf839e213be2 with result from task 878
parsl.dataflow.dflow DEBUG: Task 878 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/e6d8b20c-6ffa-4c93-94e2-34edd4ec0a4f_calibrate_855909_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/e6d8b20c-6ffa-4c93-94e2-34edd4ec0a4f_calibrate_855909_30.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:33 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:33 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 593 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 657d460d21b4bde84c79e5bd64d1646a with result from task 593
parsl.dataflow.dflow DEBUG: Task 593 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/194cd80d-2c83-465c-b668-fe790ad27c7b_calibrate_855820_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/194cd80d-2c83-465c-b668-fe790ad27c7b_calibrate_855820_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:33 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9c25a4ce2f96c590314c552d34d94b56 with result from task 220
parsl.dataflow.dflow DEBUG: Task 220 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 221 has memoization hash 3870db251893260bd462350f205843ee
parsl.dataflow.memoization INFO: Task 221 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a66de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 1007
parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/3fd83976-c949-48d3-a9e6-af1ecfb422a9_calibrate_855822_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/3fd83976-c949-48d3-a9e6-af1ecfb422a9_calibrate_855822_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/bc8324a4-3014-4ca8-a5a1-34ff9e687ac7_characterizeImage_855822_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/bc8324a4-3014-4ca8-a5a1-34ff9e687ac7_characterizeImage_855822_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:33 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:34 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:34 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:34 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:34 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:35 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:35 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:35 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:35 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 611 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0e1efb8fc40dc9ac9d24b908a515d67b with result from task 611
parsl.dataflow.dflow DEBUG: Task 611 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/74c19f27-7242-4092-bc7a-61af040398eb_calibrate_855820_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/74c19f27-7242-4092-bc7a-61af040398eb_calibrate_855820_20.stderr
parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e904e688924c40b988c4473675d4c75f with result from task 166
parsl.dataflow.dflow DEBUG: Task 166 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 355004781cacdcb133301495d03d3877
parsl.dataflow.memoization INFO: Task 167 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a43560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 1008
parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/3a79a315-d0ad-4906-a080-c0f3e957db7d_calibrate_855822_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/3a79a315-d0ad-4906-a080-c0f3e957db7d_calibrate_855822_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/d60acd2b-0de2-4151-8a8c-3093be5bba4c_characterizeImage_855822_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/d60acd2b-0de2-4151-8a8c-3093be5bba4c_characterizeImage_855822_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:38 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:38 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.dataflow.dflow INFO: Task 857 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 08bb0d9cda6607c88dd36a412870fa72 with result from task 857
parsl.dataflow.dflow DEBUG: Task 857 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/78e63937-f7b6-4e11-9433-d785733cb762_calibrate_855820_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/78e63937-f7b6-4e11-9433-d785733cb762_calibrate_855820_35.stderr
parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9ae71d5c76775b5a9b30f2124ef68db4 with result from task 316
parsl.dataflow.dflow DEBUG: Task 316 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 317 has memoization hash 87df33c52d32e2506dd9f74cd30748d7
parsl.dataflow.memoization INFO: Task 317 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 1009
parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/c6442850-468a-4c9c-94a9-e7b526acf47c_calibrate_855822_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/c6442850-468a-4c9c-94a9-e7b526acf47c_calibrate_855822_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/d12e0494-90db-4918-b785-528f128fdaa3_characterizeImage_855822_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/d12e0494-90db-4918-b785-528f128fdaa3_characterizeImage_855822_1.stderr
parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1d4248a30b44b976b8ce524b79cf2630 with result from task 161
parsl.dataflow.dflow DEBUG: Task 161 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/918df869-b3d1-47af-bd35-ff9de151fa66_calibrate_855820_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/918df869-b3d1-47af-bd35-ff9de151fa66_calibrate_855820_1.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:38 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.dataflow.dflow INFO: Task 413 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7036c557c6881346572f3b40d4c512b3 with result from task 413
parsl.dataflow.dflow DEBUG: Task 413 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/e306d769-6d1e-462a-9a5b-709e53c845a6_calibrate_855820_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/e306d769-6d1e-462a-9a5b-709e53c845a6_calibrate_855820_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:39 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:39 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:39 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:39 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:40 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:40 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:40 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:40 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d91711a897ed3d13d458eb984d243c67 with result from task 218
parsl.dataflow.dflow DEBUG: Task 218 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/c5362943-c05b-47f6-bf34-12e0a1fbb5aa_calibrate_855820_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/c5362943-c05b-47f6-bf34-12e0a1fbb5aa_calibrate_855820_13.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:41 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:41 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:41 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:41 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:42 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:42 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:42 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:42 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:43 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:43 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:43 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:43 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:44 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:44 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:44 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:44 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 901 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 11f5102fdeb90e7c6f8b23e51c3af710 with result from task 901
parsl.dataflow.dflow DEBUG: Task 901 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 902 has memoization hash 37f04b408c7fc1d5e5f144d83074386b
parsl.dataflow.memoization INFO: Task 902 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156eb600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 902 try 0 launched on executor multi with executor id 1010
parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/cca711f2-1e49-4a53-a028-0a7027af994a_calibrate_855821_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/cca711f2-1e49-4a53-a028-0a7027af994a_calibrate_855821_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/0002dd9d-719a-4f3a-a3c4-5ad2824dc8e6_characterizeImage_855821_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/0002dd9d-719a-4f3a-a3c4-5ad2824dc8e6_characterizeImage_855821_39.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:45 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 433 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2e7fffb5a14c6d781eac3540286967fc with result from task 433
parsl.dataflow.dflow DEBUG: Task 433 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 434 has memoization hash 1a327002471d38f706e60f13b49a8929
parsl.dataflow.memoization INFO: Task 434 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 434 try 0 launched on executor multi with executor id 1011
parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/214e4f66-8c1b-45d2-ad7c-5276f3c2eba5_calibrate_855821_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/214e4f66-8c1b-45d2-ad7c-5276f3c2eba5_calibrate_855821_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/8e1ac0e1-a94d-457a-91d1-a2be127f1a13_characterizeImage_855821_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/8e1ac0e1-a94d-457a-91d1-a2be127f1a13_characterizeImage_855821_20.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:45 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:45 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:46 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:46 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5b152b1eeb38cdbcff24e57446f75e3d with result from task 244
parsl.dataflow.dflow DEBUG: Task 244 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 1baaf62d3b81c9e0450b764c5f9e9450
parsl.dataflow.memoization INFO: Task 245 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 1012
parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/5b64dde7-dd73-483e-aaa7-e387d9e0ea96_calibrate_855820_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/5b64dde7-dd73-483e-aaa7-e387d9e0ea96_calibrate_855820_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/41534af1-32a6-43bf-b993-c06516b8dd3c_characterizeImage_855820_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/41534af1-32a6-43bf-b993-c06516b8dd3c_characterizeImage_855820_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:46 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 643 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d8f5a9e56d413e290af63d6228ed3e6e with result from task 643
parsl.dataflow.dflow DEBUG: Task 643 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 644 has memoization hash f10b63ced8231be7f95ae50b64abd647
parsl.dataflow.memoization INFO: Task 644 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 644 try 0 launched on executor multi with executor id 1013
parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/b18b9a5f-0928-40b4-a475-53694ab297f3_calibrate_855822_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/b18b9a5f-0928-40b4-a475-53694ab297f3_calibrate_855822_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/62f0adef-d30a-4385-a601-fac321ed6719_characterizeImage_855822_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/62f0adef-d30a-4385-a601-fac321ed6719_characterizeImage_855822_45.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:46 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:47 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:47 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 403 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4e11f6cc38ba088acb1f87ae4c64ce91 with result from task 403
parsl.dataflow.dflow DEBUG: Task 403 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 404 has memoization hash 3b06e1557724b82d051a141c761a3577
parsl.dataflow.memoization INFO: Task 404 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 1014
parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/390a908d-86cd-4dca-8be9-7667af82a98e_calibrate_855821_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/390a908d-86cd-4dca-8be9-7667af82a98e_calibrate_855821_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/763a5823-d0dd-40d9-b6ea-3b68059c081b_characterizeImage_855821_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/763a5823-d0dd-40d9-b6ea-3b68059c081b_characterizeImage_855821_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:47 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:47 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 490 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 602f531ac7efb4981ef8c7009926e569 with result from task 490
parsl.dataflow.dflow DEBUG: Task 490 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 491 has memoization hash c2d3b18b31485bdb458273ec74804a75
parsl.dataflow.memoization INFO: Task 491 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15944860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 491 try 0 launched on executor multi with executor id 1015
parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/ff879527-4f7f-4400-b240-2843b38b4517_calibrate_855821_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/ff879527-4f7f-4400-b240-2843b38b4517_calibrate_855821_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/c980a735-aac0-4722-a98e-f4943b00848c_characterizeImage_855821_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/c980a735-aac0-4722-a98e-f4943b00848c_characterizeImage_855821_3.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:48 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:48 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1030 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 09fca4ab4cfd5af6df93d128df70ebe1 with result from task 1030
parsl.dataflow.dflow DEBUG: Task 1030 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1031 has memoization hash ecad41b48309fba277acd916cc2cd3ac
parsl.dataflow.memoization INFO: Task 1031 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1031 try 0 launched on executor multi with executor id 1016
parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/7fb9c1b8-6a04-4ced-ab27-b5f12e664ff8_calibrate_855821_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/7fb9c1b8-6a04-4ced-ab27-b5f12e664ff8_calibrate_855821_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/64067b22-a739-4683-989a-e74110f922a6_characterizeImage_855821_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/64067b22-a739-4683-989a-e74110f922a6_characterizeImage_855821_19.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:48 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:49 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:49 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:49 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:49 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:50 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:50 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 626 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2c555e2deef663235f3918ba0018fdf7 with result from task 626
parsl.dataflow.dflow DEBUG: Task 626 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/2b9f40ff-6a49-48ed-adcb-59bb495d3741_calibrate_855820_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/2b9f40ff-6a49-48ed-adcb-59bb495d3741_calibrate_855820_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:50 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:51 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:51 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:51 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:51 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:52 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:52 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:52 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 410d98db872ea9d281e0c325151f1f69 with result from task 269
parsl.dataflow.dflow DEBUG: Task 269 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7fad7af5-84c8-4d41-8855-9542048ebf8b_calibrate_855820_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7fad7af5-84c8-4d41-8855-9542048ebf8b_calibrate_855820_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:53 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:53 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 977 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b8e95816646dba584d7a8655d670c5e6 with result from task 977
parsl.dataflow.dflow DEBUG: Task 977 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/caee272b-6a8c-422b-8c00-cf71813df75e_calibrate_855820_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/caee272b-6a8c-422b-8c00-cf71813df75e_calibrate_855820_46.stderr
parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e65a926e2546ddee1fcfbd7a42239eaf with result from task 352
parsl.dataflow.dflow DEBUG: Task 352 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 353 has memoization hash e0d2cf545999cae267e7485b97bbcc7e
parsl.dataflow.memoization INFO: Task 353 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16abb7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 1017
parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/16142cd8-ad3d-4f37-869d-3dbc0627c553_calibrate_855821_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/16142cd8-ad3d-4f37-869d-3dbc0627c553_calibrate_855821_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/f86520ac-ffae-4999-873d-215f504c9fa4_characterizeImage_855821_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/f86520ac-ffae-4999-873d-215f504c9fa4_characterizeImage_855821_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:53 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:54 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:54 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:54 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:58:55 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 162
parsl.dataflow.strategy DEBUG: Executor multi has 162 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 162
parsl.dataflow.strategy DEBUG: Executor multi has 162 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.dflow INFO: Task 734 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 757393eb81961e41bb661ca4d5c42c73 with result from task 734
parsl.dataflow.dflow DEBUG: Task 734 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/a62d0d07-e493-4824-ae7c-8295fe3b8776_calibrate_855820_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/a62d0d07-e493-4824-ae7c-8295fe3b8776_calibrate_855820_30.stderr
parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 883720ff63528a288330fca463aecd47 with result from task 25
parsl.dataflow.dflow DEBUG: Task 25 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 8cf0336645e9b9d64523a7d198234f2a
parsl.dataflow.memoization INFO: Task 26 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169ff100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 1018
parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/22cdaa20-484f-4cf6-b076-19598f61ed07_calibrate_855822_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/22cdaa20-484f-4cf6-b076-19598f61ed07_calibrate_855822_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/fc869b7e-88d6-4c01-b3d5-35d5d46b07ad_characterizeImage_855822_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/fc869b7e-88d6-4c01-b3d5-35d5d46b07ad_characterizeImage_855822_9.stderr
parsl.dataflow.dflow INFO: Task 1094 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5e92f0c0b144917fc49533ad9edae2de with result from task 1094
parsl.dataflow.dflow DEBUG: Task 1094 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/4d8d543f-3cb4-4bde-8311-ebe18d9ee388_calibrate_855820_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/4d8d543f-3cb4-4bde-8311-ebe18d9ee388_calibrate_855820_4.stderr
parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 65202395c5ad350c78c3dab603060924 with result from task 340
parsl.dataflow.dflow DEBUG: Task 340 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 341 has memoization hash 405cf414ab96d42e2b15581da218aeb9
parsl.dataflow.memoization INFO: Task 341 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16abab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 1019
parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/b7d6d7a1-5bc6-442c-a65e-33a807b51bea_calibrate_855821_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/b7d6d7a1-5bc6-442c-a65e-33a807b51bea_calibrate_855821_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/a3bc10e0-15bf-47b1-945d-2986331c7274_characterizeImage_855821_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/a3bc10e0-15bf-47b1-945d-2986331c7274_characterizeImage_855821_13.stderr
parsl.dataflow.dflow INFO: Task 1114 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1de023253abcf145d857ef1bdc44cd3b with result from task 1114
parsl.dataflow.dflow DEBUG: Task 1114 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1115 has memoization hash 70cb3710309791d2e01d705bd8099a4b
parsl.dataflow.memoization INFO: Task 1115 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1560d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1115 try 0 launched on executor multi with executor id 1020
parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/1c47251c-2f68-446c-8341-444c6c8b4add_calibrate_855822_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/1c47251c-2f68-446c-8341-444c6c8b4add_calibrate_855822_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/2650a97b-d793-4f1a-a4f8-d3640c0d5e6d_characterizeImage_855822_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/2650a97b-d793-4f1a-a4f8-d3640c0d5e6d_characterizeImage_855822_14.stderr
2024-03-09 20:59:00 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 20:59:00 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 890 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8e4e95439ea526c5108fcfab58c4b3af with result from task 890
parsl.dataflow.dflow DEBUG: Task 890 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/13bc01c6-7c4e-4cd8-81d9-2b1dd54475b4_calibrate_855820_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/13bc01c6-7c4e-4cd8-81d9-2b1dd54475b4_calibrate_855820_3.stderr
parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0099714152e29b9f631a874cbe4662c3 with result from task 47
parsl.dataflow.dflow DEBUG: Task 47 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/453b19ad-592b-445f-bb91-7441b49a4a45_calibrate_855820_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/453b19ad-592b-445f-bb91-7441b49a4a45_calibrate_855820_27.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:00 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:01 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:01 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:01 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 30321e8cbaca630e68ac26ad710a397c with result from task 83
parsl.dataflow.dflow DEBUG: Task 83 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/eac7ad90-585d-4b21-9141-353e09984a41_calibrate_855820_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/eac7ad90-585d-4b21-9141-353e09984a41_calibrate_855820_11.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:01 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:02 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:02 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:02 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 424 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c8b5f8c007053702a7087546998e82a3 with result from task 424
parsl.dataflow.dflow DEBUG: Task 424 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 425 has memoization hash 651d2671989b7f6dadc4b9a64a248d8f
parsl.dataflow.memoization INFO: Task 425 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 1021
parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/8dc435b4-69bd-413c-afaf-688fa44e4282_calibrate_855821_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/8dc435b4-69bd-413c-afaf-688fa44e4282_calibrate_855821_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/772062d8-8c67-4a07-9acf-af46db68f112_characterizeImage_855821_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/772062d8-8c67-4a07-9acf-af46db68f112_characterizeImage_855821_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:03 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:03 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 445 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 28a01e50ac1a87c0285a631611ca328b with result from task 445
parsl.dataflow.dflow DEBUG: Task 445 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 446 has memoization hash cecc914c1a08955ac1def65633cb8877
parsl.dataflow.memoization INFO: Task 446 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 446 try 0 launched on executor multi with executor id 1022
parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/5682b7f5-433f-422c-9e61-f6cab04b2dd0_calibrate_855821_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/5682b7f5-433f-422c-9e61-f6cab04b2dd0_calibrate_855821_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ddf62b0d-6d11-4bd3-a5b9-475c51c7b346_characterizeImage_855821_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ddf62b0d-6d11-4bd3-a5b9-475c51c7b346_characterizeImage_855821_10.stderr
parsl.dataflow.dflow INFO: Task 400 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d02a813f9a20fff9d7387d89346a53a4 with result from task 400
parsl.dataflow.dflow DEBUG: Task 400 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 401 has memoization hash f7f4b16239d05e46b3ca634a91fa0463
parsl.dataflow.memoization INFO: Task 401 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 1023
parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/667c25f4-5fe7-4dcd-8e60-7ac5f417fc47_calibrate_855822_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/667c25f4-5fe7-4dcd-8e60-7ac5f417fc47_calibrate_855822_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/0acbe00e-077f-40d6-b46a-c81f36470ac7_characterizeImage_855822_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/0acbe00e-077f-40d6-b46a-c81f36470ac7_characterizeImage_855822_40.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:03 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 1076 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fb3104bd6b04ccda83cc8ccde48bbd22 with result from task 1076
parsl.dataflow.dflow DEBUG: Task 1076 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/b8a91176-3047-49d9-9a10-af57fccfd54c_calibrate_855820_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/b8a91176-3047-49d9-9a10-af57fccfd54c_calibrate_855820_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:04 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:04 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:04 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 896 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a62bd0367c8a32945d5e43ef9bb667be with result from task 896
parsl.dataflow.dflow DEBUG: Task 896 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/8a36815c-2b22-484a-be85-6a72b3a4b74b_calibrate_855820_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/8a36815c-2b22-484a-be85-6a72b3a4b74b_calibrate_855820_60.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:04 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ff2d1a10240978946e6b1ec406e8d745 with result from task 301
parsl.dataflow.dflow DEBUG: Task 301 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 302 has memoization hash d9797ff3b14cce5933db53d1df8fc5de
parsl.dataflow.memoization INFO: Task 302 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16ab82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 1024
parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f84fb7a1-7df0-433e-a39c-47a874fac8ab_calibrate_855821_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f84fb7a1-7df0-433e-a39c-47a874fac8ab_calibrate_855821_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/a8250aee-2c03-4096-ac18-2a3ba4949160_characterizeImage_855821_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/a8250aee-2c03-4096-ac18-2a3ba4949160_characterizeImage_855821_15.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:05 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:05 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 584 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6640d994d2a709d08b484d7270f69047 with result from task 584
parsl.dataflow.dflow DEBUG: Task 584 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/e9e2fb6b-bef2-4aed-b67b-92f7c1e7b14d_calibrate_855820_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/e9e2fb6b-bef2-4aed-b67b-92f7c1e7b14d_calibrate_855820_48.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:05 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:06 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 26981d5566435bb10904208869ba3e6d with result from task 229
parsl.dataflow.dflow DEBUG: Task 229 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 230 has memoization hash f9fe5af4bcaf595da6218b5b08309f04
parsl.dataflow.memoization INFO: Task 230 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a67740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 1025
parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/a48b7ec2-3997-4c09-b646-5ed53f367e59_calibrate_855822_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/a48b7ec2-3997-4c09-b646-5ed53f367e59_calibrate_855822_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/64ab648d-e9a3-4190-b3f3-5d3a6cf90256_characterizeImage_855822_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/64ab648d-e9a3-4190-b3f3-5d3a6cf90256_characterizeImage_855822_39.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:06 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 632 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b9cbcccc5d89340f5ee218eec5686f93 with result from task 632
parsl.dataflow.dflow DEBUG: Task 632 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/e686b97d-f026-4bd6-aa8f-87d6f6a104f3_calibrate_855820_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/e686b97d-f026-4bd6-aa8f-87d6f6a104f3_calibrate_855820_8.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:06 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:06 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 467 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 584a36f9bc0d2d5deb3ccd4a2c00615b with result from task 467
parsl.dataflow.dflow DEBUG: Task 467 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/a24e0007-a778-40e0-ad80-98a39d476a1a_calibrate_855820_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/a24e0007-a778-40e0-ad80-98a39d476a1a_calibrate_855820_16.stderr
parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5ab1c7508c6beea68d0ebc50b79140ce with result from task 215
parsl.dataflow.dflow DEBUG: Task 215 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/cf674d0e-1649-46a2-8543-1fc4795fbad0_calibrate_855820_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/cf674d0e-1649-46a2-8543-1fc4795fbad0_calibrate_855820_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:07 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:07 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5561f32e6f4b4b89ef1adb755cd928ff with result from task 235
parsl.dataflow.dflow DEBUG: Task 235 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 236 has memoization hash 448969340e2e9d1787291ab06f674e46
parsl.dataflow.memoization INFO: Task 236 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a67d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 1026
parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/38cb4ed5-5cfa-4d34-b98c-e9355f9bc5a8_calibrate_855821_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/38cb4ed5-5cfa-4d34-b98c-e9355f9bc5a8_calibrate_855821_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/c1f3fa98-d0bf-4f86-b700-ad0c23e1fd85_characterizeImage_855821_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/c1f3fa98-d0bf-4f86-b700-ad0c23e1fd85_characterizeImage_855821_31.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:07 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:08 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:08 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:08 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 973 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 79effe03b3acbf8ababb7147bb9997aa with result from task 973
parsl.dataflow.dflow DEBUG: Task 973 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 974 has memoization hash 369d56df2939e4e236df3c176fea0af3
parsl.dataflow.memoization INFO: Task 974 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 974 try 0 launched on executor multi with executor id 1027
parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f250d9ed-ab5c-406b-9278-17f05c630f8c_calibrate_855821_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f250d9ed-ab5c-406b-9278-17f05c630f8c_calibrate_855821_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/2952b10e-3da7-4654-b616-2753c65d6f46_characterizeImage_855821_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/2952b10e-3da7-4654-b616-2753c65d6f46_characterizeImage_855821_5.stderr
parsl.dataflow.dflow INFO: Task 818 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 33c216aa075fa4492a788b27a7c95a7f with result from task 818
parsl.dataflow.dflow DEBUG: Task 818 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/929b43f0-347e-402c-bede-0fb884df93b6_calibrate_855820_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/929b43f0-347e-402c-bede-0fb884df93b6_calibrate_855820_40.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:10 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:11 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.dataflow.dflow INFO: Task 871 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fef774e2c038f71ef37c60fc6a3b7190 with result from task 871
parsl.dataflow.dflow DEBUG: Task 871 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 872 has memoization hash 869c6921f656b3eb185315f8cd2e26c1
parsl.dataflow.memoization INFO: Task 872 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 872 try 0 launched on executor multi with executor id 1028
parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/021c6d30-7156-4ae0-a9a7-0063601c10f5_calibrate_855822_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/021c6d30-7156-4ae0-a9a7-0063601c10f5_calibrate_855822_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a3fa3bf8-5523-4fc9-adb3-19d427cb862a_characterizeImage_855822_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a3fa3bf8-5523-4fc9-adb3-19d427cb862a_characterizeImage_855822_32.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:11 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:12 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:12 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:12 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:13 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:13 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:13 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:13 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8396ad2e99959dfa4bf126bf9c5daa6b with result from task 140
parsl.dataflow.dflow DEBUG: Task 140 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/60de52e6-14bb-4755-a6f6-98f1d908801c_calibrate_855820_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/60de52e6-14bb-4755-a6f6-98f1d908801c_calibrate_855820_47.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:14 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6379d4913b177dee195cdbd2309ecf9c with result from task 178
parsl.dataflow.dflow DEBUG: Task 178 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 8c8dec3e213f35f20591e17c387264bc
parsl.dataflow.memoization INFO: Task 179 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a64220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 1029
parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/cf7e54fa-1a7b-41b2-a4f6-6ece4bcb1148_calibrate_855821_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/cf7e54fa-1a7b-41b2-a4f6-6ece4bcb1148_calibrate_855821_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/712ce072-1354-4115-8b52-572c14a129cd_characterizeImage_855821_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/712ce072-1354-4115-8b52-572c14a129cd_characterizeImage_855821_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:14 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:14 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:15 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f973ad4dfaef6c726ae0baded3d4e609 with result from task 226
parsl.dataflow.dflow DEBUG: Task 226 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 227 has memoization hash a9bce1c8c219e8301551e107d116b838
parsl.dataflow.memoization INFO: Task 227 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a67420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 1030
parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/53cd8091-bd69-41d3-96bb-7bc78b9e3bdd_calibrate_855821_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/53cd8091-bd69-41d3-96bb-7bc78b9e3bdd_calibrate_855821_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/eb151e5f-ad89-4460-b1d9-28231e3a34fd_characterizeImage_855821_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/eb151e5f-ad89-4460-b1d9-28231e3a34fd_characterizeImage_855821_33.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:15 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 421 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 001043b7ded6bce21265d32f56a0c359 with result from task 421
parsl.dataflow.dflow DEBUG: Task 421 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 422 has memoization hash b63709df4ec3763f84e2275b44d3b939
parsl.dataflow.memoization INFO: Task 422 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 422 try 0 launched on executor multi with executor id 1031
parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/2192cb69-a0e2-4e0f-8c1e-740b533904a7_calibrate_855821_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/2192cb69-a0e2-4e0f-8c1e-740b533904a7_calibrate_855821_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ad1108bd-1043-4d78-93cf-c556eb997ffc_characterizeImage_855821_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ad1108bd-1043-4d78-93cf-c556eb997ffc_characterizeImage_855821_35.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:15 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 77a2de79be55f7c38e2c4686779bfc40 with result from task 250
parsl.dataflow.dflow DEBUG: Task 250 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 251 has memoization hash 818870c1c768f0b93651cb178764d8f6
parsl.dataflow.memoization INFO: Task 251 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 1032
parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/32fb4016-a4cc-4036-8202-576add071d07_calibrate_855822_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/32fb4016-a4cc-4036-8202-576add071d07_calibrate_855822_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/dadb0dfe-a789-4d3d-87dc-8734e7364f25_characterizeImage_855822_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/dadb0dfe-a789-4d3d-87dc-8734e7364f25_characterizeImage_855822_15.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:16 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:16 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry afdca71199062635bf45c1e71cbafa91 with result from task 115
parsl.dataflow.dflow DEBUG: Task 115 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 9c0ebad85b276f5d9bfe9f05fa9f969d
parsl.dataflow.memoization INFO: Task 116 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a40220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 1033
parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/bb716244-b9c3-4a8c-86de-28ba123f9c5c_calibrate_855821_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/bb716244-b9c3-4a8c-86de-28ba123f9c5c_calibrate_855821_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/cfe6de47-9909-43e3-a943-025980015432_characterizeImage_855821_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/cfe6de47-9909-43e3-a943-025980015432_characterizeImage_855821_32.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:16 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:16 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 707 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0741bb203a046cda5791247396622561 with result from task 707
parsl.dataflow.dflow DEBUG: Task 707 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/b9eed9a4-44b7-4164-b7b2-fff9a4ec1f61_calibrate_855820_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/b9eed9a4-44b7-4164-b7b2-fff9a4ec1f61_calibrate_855820_23.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:17 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:17 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:17 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:18 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:18 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 544 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a03b06f450d9f791f7429cd822e28a48 with result from task 544
parsl.dataflow.dflow DEBUG: Task 544 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 545 has memoization hash cc75cec9ed896257b833f540fbf4ed9b
parsl.dataflow.memoization INFO: Task 545 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 545 try 0 launched on executor multi with executor id 1034
parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/a6766b0b-56c6-40c5-bd05-76f2da1960bc_calibrate_855822_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/a6766b0b-56c6-40c5-bd05-76f2da1960bc_calibrate_855822_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/2cbf3111-8af8-48cc-b869-ddf933599d2e_characterizeImage_855822_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/2cbf3111-8af8-48cc-b869-ddf933599d2e_characterizeImage_855822_3.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:18 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:19 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:19 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 479 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5f0bb4892740ce3973a70afe549fefb9 with result from task 479
parsl.dataflow.dflow DEBUG: Task 479 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/5f60bd31-69ac-4864-b7a4-f8c12fce3f2c_calibrate_855820_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/5f60bd31-69ac-4864-b7a4-f8c12fce3f2c_calibrate_855820_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:19 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 518 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3866a4b19335e7bfb80e44f68d686fc1 with result from task 518
parsl.dataflow.dflow DEBUG: Task 518 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/4de6f49d-c2f6-40ee-8ee3-1e2d880cb1d8_calibrate_855820_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/4de6f49d-c2f6-40ee-8ee3-1e2d880cb1d8_calibrate_855820_38.stderr
parsl.dataflow.dflow INFO: Task 700 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0f12d4053081539917dc37272cde2724 with result from task 700
parsl.dataflow.dflow DEBUG: Task 700 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 701 has memoization hash f668af1d02503b2f85e075c994e4eee5
parsl.dataflow.memoization INFO: Task 701 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 701 try 0 launched on executor multi with executor id 1035
parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/fcbc7466-e3a2-4afa-944f-76264fdc060b_calibrate_855821_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/fcbc7466-e3a2-4afa-944f-76264fdc060b_calibrate_855821_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/de38ec18-22eb-43af-ac56-74b362e327c4_characterizeImage_855821_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/de38ec18-22eb-43af-ac56-74b362e327c4_characterizeImage_855821_45.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:20 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:20 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:20 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:20 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d4bb03926925e669bb7f6b32f6271d82 with result from task 31
parsl.dataflow.dflow DEBUG: Task 31 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 32 has memoization hash de465bd65ddf630660d67c427b14aa2e
parsl.dataflow.memoization INFO: Task 32 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169feac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 1036
parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/a8fc75fe-554a-4aab-9ab5-3da9ee82e81c_calibrate_855822_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/a8fc75fe-554a-4aab-9ab5-3da9ee82e81c_calibrate_855822_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/6bc69d38-1305-453b-a53c-54929b7e9534_characterizeImage_855822_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/6bc69d38-1305-453b-a53c-54929b7e9534_characterizeImage_855822_8.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:21 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 748 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 778fb66115ffc30802009627f4f4ac7b with result from task 748
parsl.dataflow.dflow DEBUG: Task 748 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 749 has memoization hash 7c11738386018701249836cb58b1f535
parsl.dataflow.memoization INFO: Task 749 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15859620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 749 try 0 launched on executor multi with executor id 1037
parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/d1e992de-1295-4fac-ae36-fe7149cc179e_calibrate_855821_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/d1e992de-1295-4fac-ae36-fe7149cc179e_calibrate_855821_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/6f155b08-1a87-4394-886d-d48ed41b770e_characterizeImage_855821_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/6f155b08-1a87-4394-886d-d48ed41b770e_characterizeImage_855821_22.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:21 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1033 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fd6e9c5e482e9cf9dce63e5e08a30764 with result from task 1033
parsl.dataflow.dflow DEBUG: Task 1033 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1034 has memoization hash 270b3a47e0d167936b28350e67a2899c
parsl.dataflow.memoization INFO: Task 1034 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1034 try 0 launched on executor multi with executor id 1038
parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/918ca5d8-9d7c-4700-804c-b4d3f1ac3498_calibrate_855822_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/918ca5d8-9d7c-4700-804c-b4d3f1ac3498_calibrate_855822_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/70ccf5df-2145-40f9-9777-fdb2aa7ed66f_characterizeImage_855822_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/70ccf5df-2145-40f9-9777-fdb2aa7ed66f_characterizeImage_855822_4.stderr
parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ed79657f3c87999e4aa3f54e82442918 with result from task 191
parsl.dataflow.dflow DEBUG: Task 191 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/b9becbda-b490-49d5-a68c-1c58f5b390a7_calibrate_855820_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/b9becbda-b490-49d5-a68c-1c58f5b390a7_calibrate_855820_29.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:21 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:22 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:22 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:22 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:23 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:23 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 574 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 004ab7dd17d5aab59251c0b13f3dbb86 with result from task 574
parsl.dataflow.dflow DEBUG: Task 574 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 575 has memoization hash 546322a06a3bc6274b62cbe1a30f5ab6
parsl.dataflow.memoization INFO: Task 575 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 575 try 0 launched on executor multi with executor id 1039
parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/4600d054-8fe1-4a94-abea-c0112f97724d_calibrate_855822_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/4600d054-8fe1-4a94-abea-c0112f97724d_calibrate_855822_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/0039363b-4812-4620-a5b0-0bc9d3b18d5d_characterizeImage_855822_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/0039363b-4812-4620-a5b0-0bc9d3b18d5d_characterizeImage_855822_16.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:23 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:24 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:24 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:24 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1046 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 037349b1b8dc50f4b88049829999534b with result from task 1046
parsl.dataflow.dflow DEBUG: Task 1046 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/afe75ef8-740d-40fc-bfab-b3450957e5c8_calibrate_855820_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/afe75ef8-740d-40fc-bfab-b3450957e5c8_calibrate_855820_17.stderr
parsl.dataflow.dflow INFO: Task 503 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry edc8df9403a05205124532335e949122 with result from task 503
parsl.dataflow.dflow DEBUG: Task 503 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/2453f2c0-3d23-4630-9577-6735ef9f3bad_calibrate_855820_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/2453f2c0-3d23-4630-9577-6735ef9f3bad_calibrate_855820_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:24 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 827 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7707f9c1728595bb088de4aeb02d17b4 with result from task 827
parsl.dataflow.dflow DEBUG: Task 827 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/1f22c968-ab71-49cb-a122-71242b0a8b07_calibrate_855820_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/1f22c968-ab71-49cb-a122-71242b0a8b07_calibrate_855820_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:25 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1052 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1b1d4e047b90f57bce30e4384c1b5aee with result from task 1052
parsl.dataflow.dflow DEBUG: Task 1052 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/f2f7ca35-31ab-42c8-b5ec-670c2dd106ff_calibrate_855820_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/f2f7ca35-31ab-42c8-b5ec-670c2dd106ff_calibrate_855820_53.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:25 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:25 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:26 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1084 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 88b6652c35afe8a5ae134781771ccd31 with result from task 1084
parsl.dataflow.dflow DEBUG: Task 1084 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1085 has memoization hash 749111005d1d855cd336811ab6bd469f
parsl.dataflow.memoization INFO: Task 1085 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1085 try 0 launched on executor multi with executor id 1040
parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/09eae4ed-c75d-48a0-9337-235b7b91ea74_calibrate_855822_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/09eae4ed-c75d-48a0-9337-235b7b91ea74_calibrate_855822_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/fc5056c5-506c-4a84-9013-750786a510ef_characterizeImage_855822_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/fc5056c5-506c-4a84-9013-750786a510ef_characterizeImage_855822_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:26 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 727 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0ea51a7dafb5b59c24c2af9e303449ee with result from task 727
parsl.dataflow.dflow DEBUG: Task 727 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 728 has memoization hash 6fecf814449aaef1537b3c17187f924d
parsl.dataflow.memoization INFO: Task 728 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15858220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 728 try 0 launched on executor multi with executor id 1041
parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/bcc5b1c7-e62a-4f08-b00c-f8bf24bb99ce_calibrate_855822_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/bcc5b1c7-e62a-4f08-b00c-f8bf24bb99ce_calibrate_855822_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/2920bbf3-d993-43d3-bd19-e7726bd85d13_characterizeImage_855822_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/2920bbf3-d993-43d3-bd19-e7726bd85d13_characterizeImage_855822_31.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:26 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:27 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1018 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f4ee73b47b8cff4e6a4c1a1987a98295 with result from task 1018
parsl.dataflow.dflow DEBUG: Task 1018 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1019 has memoization hash 254c4892a81b31a7f39ca51cb2e9f8a6
parsl.dataflow.memoization INFO: Task 1019 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1019 try 0 launched on executor multi with executor id 1042
parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/4175ceee-e704-4489-8005-7af750dff222_calibrate_855822_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/4175ceee-e704-4489-8005-7af750dff222_calibrate_855822_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a57c1a8e-c630-4041-a7b0-476fc439662a_characterizeImage_855822_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a57c1a8e-c630-4041-a7b0-476fc439662a_characterizeImage_855822_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:27 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:27 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:27 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 84c5171cde042008813819702c604184 with result from task 19
parsl.dataflow.dflow DEBUG: Task 19 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 80b77ac3b3dc963de57d1f3de2118521
parsl.dataflow.memoization INFO: Task 20 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 1043
parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/d93ff072-bd90-4756-95ba-28ee1544ccee_calibrate_855822_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/d93ff072-bd90-4756-95ba-28ee1544ccee_calibrate_855822_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/b9dcbc31-bbba-4201-a5ca-aa76b4064b54_characterizeImage_855822_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/b9dcbc31-bbba-4201-a5ca-aa76b4064b54_characterizeImage_855822_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:28 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:28 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:28 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 911 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 306ecadc4f355aedd25912bec5488989 with result from task 911
parsl.dataflow.dflow DEBUG: Task 911 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7e7fa924-166d-4f1d-aa0f-cc678bbcce90_calibrate_855820_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7e7fa924-166d-4f1d-aa0f-cc678bbcce90_calibrate_855820_34.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:29 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 775 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cf5a7c90680a16a91f7cb0fd261fd308 with result from task 775
parsl.dataflow.dflow DEBUG: Task 775 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 776 has memoization hash 4a6894c0dfe456194b3621cedf8b623a
parsl.dataflow.memoization INFO: Task 776 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 776 try 0 launched on executor multi with executor id 1044
parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/76828458-18a5-4282-8838-a8b21318cf05_calibrate_855822_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/76828458-18a5-4282-8838-a8b21318cf05_calibrate_855822_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/e98d6a3d-3cb7-4a89-a258-3c13a73e7e52_characterizeImage_855822_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/e98d6a3d-3cb7-4a89-a258-3c13a73e7e52_characterizeImage_855822_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:29 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:29 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 686 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 949d221f3978246395ed7c263fdc4532 with result from task 686
parsl.dataflow.dflow DEBUG: Task 686 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/8fafc276-ba8a-4ea2-b48a-8f3c37b474f5_calibrate_855820_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/8fafc276-ba8a-4ea2-b48a-8f3c37b474f5_calibrate_855820_7.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:30 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e5eb05936eb1ef2b3e5e712f321ec301 with result from task 71
parsl.dataflow.dflow DEBUG: Task 71 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/3e4d57fb-64d6-4e67-a6d5-6976a60fb13d_calibrate_855909_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/3e4d57fb-64d6-4e67-a6d5-6976a60fb13d_calibrate_855909_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 138
parsl.dataflow.strategy DEBUG: Executor multi has 138 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 20:59:30 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 20:59:31 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:31 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:32 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:32 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:33 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7b84f213a8135c1c93935a5ec379c857 with result from task 193
parsl.dataflow.dflow DEBUG: Task 193 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 194 has memoization hash 06a50bd6d8d32657c9f4ae906eeddadf
parsl.dataflow.memoization INFO: Task 194 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a651c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 1045
parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/cf24b0f2-7ec7-4496-bbbe-98485f062929_calibrate_855822_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/cf24b0f2-7ec7-4496-bbbe-98485f062929_calibrate_855822_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/83efddc5-f05c-4a6b-8c39-408e482bcd01_characterizeImage_855822_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/83efddc5-f05c-4a6b-8c39-408e482bcd01_characterizeImage_855822_60.stderr
parsl.dataflow.dflow INFO: Task 508 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e471830c5441b3764b66123861fa01c4 with result from task 508
parsl.dataflow.dflow DEBUG: Task 508 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 509 has memoization hash 1efb8fe8b7f3835dcfd97c2f493bcc5a
parsl.dataflow.memoization INFO: Task 509 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15945b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 509 try 0 launched on executor multi with executor id 1046
parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/b26af93f-3049-482c-afbd-200c54ceaefb_calibrate_855822_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/b26af93f-3049-482c-afbd-200c54ceaefb_calibrate_855822_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/c8fed829-a540-458f-9c63-db2f23d6a6fa_characterizeImage_855822_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/c8fed829-a540-458f-9c63-db2f23d6a6fa_characterizeImage_855822_17.stderr
parsl.dataflow.dflow INFO: Task 814 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 588483d56ff79181df4bde4a61324f48 with result from task 814
parsl.dataflow.dflow DEBUG: Task 814 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 815 has memoization hash b9b87266e39c87574dc77bb4180072be
parsl.dataflow.memoization INFO: Task 815 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15895b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 815 try 0 launched on executor multi with executor id 1047
parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/6fa3f0b6-7a83-4ac8-b9ee-41cd0a3f0a03_calibrate_855821_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/6fa3f0b6-7a83-4ac8-b9ee-41cd0a3f0a03_calibrate_855821_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/4b4873fc-b96f-490b-a882-0d8ddedf4296_characterizeImage_855821_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/4b4873fc-b96f-490b-a882-0d8ddedf4296_characterizeImage_855821_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:34 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:35 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:35 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b2a814da0e5557dbab2e9f18196977b6 with result from task 124
parsl.dataflow.dflow DEBUG: Task 124 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 08bd387dcdd70d543a0aa01dc4508209
parsl.dataflow.memoization INFO: Task 125 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a409a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 1048
parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/13648ddc-cad1-4090-879a-d0b11e750e50_calibrate_855822_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/13648ddc-cad1-4090-879a-d0b11e750e50_calibrate_855822_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/e738178b-8a53-4006-a114-92a65e61b365_characterizeImage_855822_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/e738178b-8a53-4006-a114-92a65e61b365_characterizeImage_855822_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:36 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 719 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 19040efc82b8f1f532306f6f4541c618 with result from task 719
parsl.dataflow.dflow DEBUG: Task 719 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/61bb2b01-9aa7-4e4b-8205-ec8cd15de682_calibrate_855820_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/61bb2b01-9aa7-4e4b-8205-ec8cd15de682_calibrate_855820_41.stderr
parsl.dataflow.dflow INFO: Task 598 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5da7418a1d3e39241a277cee563ba437 with result from task 598
parsl.dataflow.dflow DEBUG: Task 598 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 599 has memoization hash 4313f5b88be1190a8b240aa65e16dccf
parsl.dataflow.memoization INFO: Task 599 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 599 try 0 launched on executor multi with executor id 1049
parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/9cc208aa-ecdf-44c3-9e57-ecf0e5f4896b_calibrate_855821_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/9cc208aa-ecdf-44c3-9e57-ecf0e5f4896b_calibrate_855821_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/59309771-a509-4f3c-9362-62b9a2e241e7_characterizeImage_855821_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/59309771-a509-4f3c-9362-62b9a2e241e7_characterizeImage_855821_8.stderr
parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0534c268d95f66ede5a69b94273dd351 with result from task 44
parsl.dataflow.dflow DEBUG: Task 44 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/c48849aa-8a02-4d8e-bf1c-4fb90c62e4e6_calibrate_855820_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/c48849aa-8a02-4d8e-bf1c-4fb90c62e4e6_calibrate_855820_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:36 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:37 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 580 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4a2f50a6ad030e9c91932dfe894d56bc with result from task 580
parsl.dataflow.dflow DEBUG: Task 580 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 581 has memoization hash c6a1bc5a569c31562725fb2b51d9ba64
parsl.dataflow.memoization INFO: Task 581 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 581 try 0 launched on executor multi with executor id 1050
parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/a6c3b4bf-44c0-4b62-80c5-29d4aad869c8_calibrate_855822_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/a6c3b4bf-44c0-4b62-80c5-29d4aad869c8_calibrate_855822_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/5ed7c968-5683-48f2-aa3a-07b8581d799f_characterizeImage_855822_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/5ed7c968-5683-48f2-aa3a-07b8581d799f_characterizeImage_855822_51.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:37 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 535 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 75591ac7e47e88fd29051187027adb55 with result from task 535
parsl.dataflow.dflow DEBUG: Task 535 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 536 has memoization hash fee4613ad48430a1985ae376578fcc78
parsl.dataflow.memoization INFO: Task 536 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15947740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 536 try 0 launched on executor multi with executor id 1051
parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/07907354-9395-43ad-aaf6-7e3d152d0496_calibrate_855822_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/07907354-9395-43ad-aaf6-7e3d152d0496_calibrate_855822_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/01c9bbcc-a864-411c-b656-c9ebf1f23fef_characterizeImage_855822_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/01c9bbcc-a864-411c-b656-c9ebf1f23fef_characterizeImage_855822_27.stderr
parsl.dataflow.dflow INFO: Task 385 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 03c2727bacce02c71886883c1d3323ab with result from task 385
parsl.dataflow.dflow DEBUG: Task 385 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 386 has memoization hash 2f89d939019403603e8dca559d115f4d
parsl.dataflow.memoization INFO: Task 386 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 386 try 0 launched on executor multi with executor id 1052
parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/eea5d888-917e-4976-b9fa-1d40f242987d_calibrate_855821_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/eea5d888-917e-4976-b9fa-1d40f242987d_calibrate_855821_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/51d5a040-8a5d-4d8b-a9ec-8cd9c89fdc68_characterizeImage_855821_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/51d5a040-8a5d-4d8b-a9ec-8cd9c89fdc68_characterizeImage_855821_51.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:38 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 736 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4043bee07013c2fe97a08bc46fdb175e with result from task 736
parsl.dataflow.dflow DEBUG: Task 736 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 737 has memoization hash 54d147e3d1cec0780b38a9590f8b4f82
parsl.dataflow.memoization INFO: Task 737 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158589a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 737 try 0 launched on executor multi with executor id 1053
parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f6916187-8ded-497d-b56b-3deaf46ed0bc_calibrate_855821_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f6916187-8ded-497d-b56b-3deaf46ed0bc_calibrate_855821_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/6d174999-a3bd-451a-b80d-5f6e31b813ae_characterizeImage_855821_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/6d174999-a3bd-451a-b80d-5f6e31b813ae_characterizeImage_855821_27.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:38 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cd18a9f1623a030175e9dd5a868557f2 with result from task 89
parsl.dataflow.dflow DEBUG: Task 89 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/71a060e3-794c-4c0b-be11-e68a75bb42d6_calibrate_855820_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/71a060e3-794c-4c0b-be11-e68a75bb42d6_calibrate_855820_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:39 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:39 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:40 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 1091 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 79bb7f0591b21f498ee4362bde06bfeb with result from task 1091
parsl.dataflow.dflow DEBUG: Task 1091 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/3b1b5fc8-dc28-4e20-8876-0ce95cb849a2_calibrate_855820_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/3b1b5fc8-dc28-4e20-8876-0ce95cb849a2_calibrate_855820_14.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:41 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:41 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 1097 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6bf2185c3a5dec040f65acdf8ee941f7 with result from task 1097
parsl.dataflow.dflow DEBUG: Task 1097 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/5ed09bcc-77f1-492d-908b-93a5ab0ff576_calibrate_855820_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/5ed09bcc-77f1-492d-908b-93a5ab0ff576_calibrate_855820_22.stderr
parsl.dataflow.dflow INFO: Task 1037 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0592e4ee549daebc4794faa224e41ff5 with result from task 1037
parsl.dataflow.dflow DEBUG: Task 1037 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/ae1338cd-97e8-4620-aac3-428cfd99f532_calibrate_855820_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/ae1338cd-97e8-4620-aac3-428cfd99f532_calibrate_855820_57.stderr
parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fe04ec56a4904e1a32e6629d91ec2d04 with result from task 64
parsl.dataflow.dflow DEBUG: Task 64 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 65 has memoization hash d556a5474b2ce9d2ce85afc649ce26f1
parsl.dataflow.memoization INFO: Task 65 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 1054
parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/157eaef5-ade8-40d2-8080-12dddb3d8a34_calibrate_855822_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/157eaef5-ade8-40d2-8080-12dddb3d8a34_calibrate_855822_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/0e5cfb16-d2c2-488a-8796-94d135ecc930_characterizeImage_855822_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/0e5cfb16-d2c2-488a-8796-94d135ecc930_characterizeImage_855822_11.stderr
parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5d02436569fab31be142ce7d09f174e9 with result from task 280
parsl.dataflow.dflow DEBUG: Task 280 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 281 has memoization hash 1c55a737e223095eac4943e58a09a48e
parsl.dataflow.memoization INFO: Task 281 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 1055
parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/8185e7c2-a46e-4f9b-b043-0d8578a40bdb_calibrate_855821_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/8185e7c2-a46e-4f9b-b043-0d8578a40bdb_calibrate_855821_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/4347a38c-3c1a-4b9d-b7a5-03d8abc5003b_characterizeImage_855821_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/4347a38c-3c1a-4b9d-b7a5-03d8abc5003b_characterizeImage_855821_46.stderr
parsl.dataflow.dflow INFO: Task 628 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4b13bbd1d5a8735f365532899c42f585 with result from task 628
parsl.dataflow.dflow DEBUG: Task 628 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 629 has memoization hash 8841841b24f159c361d0eb83eb03e745
parsl.dataflow.memoization INFO: Task 629 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 629 try 0 launched on executor multi with executor id 1056
parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/d24b429a-01aa-4af9-a462-9e50436b5cb2_calibrate_855822_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/d24b429a-01aa-4af9-a462-9e50436b5cb2_calibrate_855822_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/3e7db2ef-b4dc-443e-920d-a65e1bc68ec7_characterizeImage_855822_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/3e7db2ef-b4dc-443e-920d-a65e1bc68ec7_characterizeImage_855822_7.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:43 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dedec706343a0484aa2790a9b39b7a92 with result from task 173
parsl.dataflow.dflow DEBUG: Task 173 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/c62d82d3-c55f-46fd-948f-c5e304d9bb89_calibrate_855820_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/c62d82d3-c55f-46fd-948f-c5e304d9bb89_calibrate_855820_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:45 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:45 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:46 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:46 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:47 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ab81aeee09894bbd19504a32c3fd4331 with result from task 1
parsl.dataflow.dflow DEBUG: Task 1 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 65bad83ade58760f07933600af3f4d3b
parsl.dataflow.memoization INFO: Task 2 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fc680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 1057
parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/54638b1b-365c-4063-969a-1a732712af66_calibrate_855822_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/54638b1b-365c-4063-969a-1a732712af66_calibrate_855822_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/d5f119b6-af11-4fa0-a7dd-94e6ecfeb938_characterizeImage_855822_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/d5f119b6-af11-4fa0-a7dd-94e6ecfeb938_characterizeImage_855822_22.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:47 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ae988a5419b2c436fd4bf2e18effe8fe with result from task 7
parsl.dataflow.dflow DEBUG: Task 7 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 79065de3b8dbd4bc011a5fa9bf99d622
parsl.dataflow.memoization INFO: Task 8 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 1058
parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/549ae9fc-8e97-4f45-98bf-55796308ba35_calibrate_855822_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/549ae9fc-8e97-4f45-98bf-55796308ba35_calibrate_855822_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/1ee2d914-0cde-4dee-b6a3-167da531173f_characterizeImage_855822_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/1ee2d914-0cde-4dee-b6a3-167da531173f_characterizeImage_855822_13.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:48 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 1082 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d24f4ee32601bb1818dc2f0cf6b61953 with result from task 1082
parsl.dataflow.dflow DEBUG: Task 1082 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7a3ed83c-d0be-47c9-b5ef-04b3881baae4_calibrate_855820_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/7a3ed83c-d0be-47c9-b5ef-04b3881baae4_calibrate_855820_49.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:49 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:49 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 485 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4c7d5d59b3994fdf4546f0e10178cd53 with result from task 485
parsl.dataflow.dflow DEBUG: Task 485 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/a12339cc-ea8e-4314-a1fb-95024fb406d8_calibrate_855820_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/a12339cc-ea8e-4314-a1fb-95024fb406d8_calibrate_855820_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:50 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:50 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:51 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:51 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2eed5588475ea93ca17ba4e3a579a5a2 with result from task 361
parsl.dataflow.dflow DEBUG: Task 361 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 362 has memoization hash a18113f21462e7367a507d2747a842b6
parsl.dataflow.memoization INFO: Task 362 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 1059
parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/5d9e038c-a18a-472e-9c97-e9b389f3ee05_calibrate_855821_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/5d9e038c-a18a-472e-9c97-e9b389f3ee05_calibrate_855821_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/5113f289-949d-4d13-9b8f-7ec2c36a885f_characterizeImage_855821_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/5113f289-949d-4d13-9b8f-7ec2c36a885f_characterizeImage_855821_7.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:52 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:53 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.dataflow.dflow INFO: Task 874 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2b83b6fb26d8ad5d25fd10096cc035eb with result from task 874
parsl.dataflow.dflow DEBUG: Task 874 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 875 has memoization hash 7dfe323fd8f9c3819922c7da445c3e43
parsl.dataflow.memoization INFO: Task 875 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 875 try 0 launched on executor multi with executor id 1060
parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/6b6c624c-871e-4408-80cd-fa9a38728faa_calibrate_855822_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/6b6c624c-871e-4408-80cd-fa9a38728faa_calibrate_855822_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/fdd3420a-55ad-4acd-b343-b131844ced09_characterizeImage_855822_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/fdd3420a-55ad-4acd-b343-b131844ced09_characterizeImage_855822_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:53 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:54 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:54 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 90a5e7a6e4fa6c5c21698fdc2261309f with result from task 148
parsl.dataflow.dflow DEBUG: Task 148 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 149 has memoization hash 012365f2e5af654349ff59f9e197da2f
parsl.dataflow.memoization INFO: Task 149 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a422a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 1061
parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/6304a29c-6516-4259-9ce7-5a8cddea13a5_calibrate_855822_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/6304a29c-6516-4259-9ce7-5a8cddea13a5_calibrate_855822_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/82b6904a-d982-4b87-ae72-5032512b470b_characterizeImage_855822_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/82b6904a-d982-4b87-ae72-5032512b470b_characterizeImage_855822_23.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:55 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:55 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:56 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:57 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:57 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:58 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:58 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7cddc0cde5cdc63aa64309ff30ecbcbb with result from task 122
parsl.dataflow.dflow DEBUG: Task 122 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/6197dd97-be2b-42f0-8c86-c79adbaf5799_calibrate_855820_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/6197dd97-be2b-42f0-8c86-c79adbaf5799_calibrate_855820_24.stderr
parsl.dataflow.dflow INFO: Task 868 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 81ac97f76a158840d137a04fc95ca40b with result from task 868
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:59 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow DEBUG: Task 868 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 869 has memoization hash ed1c3363c41be493948de563e43538fe
parsl.dataflow.memoization INFO: Task 869 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 869 try 0 launched on executor multi with executor id 1062
parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/c8572556-02a7-457d-847b-80a5af32fc1f_calibrate_855822_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/c8572556-02a7-457d-847b-80a5af32fc1f_calibrate_855822_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/6ca1be66-963a-4a4c-b7f1-d1916172e9aa_characterizeImage_855822_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/6ca1be66-963a-4a4c-b7f1-d1916172e9aa_characterizeImage_855822_58.stderr
parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5fb14ac0b1e2308b3793fda63da97574 with result from task 241
parsl.dataflow.dflow DEBUG: Task 241 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 4740821518abb94e7140f338e0317f61
parsl.dataflow.memoization INFO: Task 242 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 1063
parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/235f360e-8b80-4577-9c31-f1fa17318f13_calibrate_855822_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/235f360e-8b80-4577-9c31-f1fa17318f13_calibrate_855822_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/c7c3ea5e-8a0c-4877-a737-7010f274c97a_characterizeImage_855822_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/c7c3ea5e-8a0c-4877-a737-7010f274c97a_characterizeImage_855822_20.stderr
parsl.dataflow.dflow INFO: Task 376 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 62ed2c34fff1958fae1bd9a3bb74d2f3 with result from task 376
parsl.dataflow.dflow DEBUG: Task 376 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 377 has memoization hash c056081c46d37de121f4bc1fbbb2dc7a
parsl.dataflow.memoization INFO: Task 377 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 377 try 0 launched on executor multi with executor id 1064
parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/fcbc3290-4a2e-4b3b-8722-8427305c7e24_calibrate_855822_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/fcbc3290-4a2e-4b3b-8722-8427305c7e24_calibrate_855822_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/b261b79e-ae50-40da-a9a2-759838909c11_characterizeImage_855822_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/b261b79e-ae50-40da-a9a2-759838909c11_characterizeImage_855822_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 20:59:59 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 452 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4b40b734eb8913f615ff6cb91be331e4 with result from task 452
parsl.dataflow.dflow DEBUG: Task 452 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/eff86f87-e2db-404b-bb28-3ceb24e4242d_calibrate_855820_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/eff86f87-e2db-404b-bb28-3ceb24e4242d_calibrate_855820_36.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:00 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:01 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 662 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e8fe1226f6dec28c93bb64cca8483c0f with result from task 662
parsl.dataflow.dflow DEBUG: Task 662 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/9df68c8d-9727-4ba4-95c0-4bfe41ae74df_calibrate_855820_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/9df68c8d-9727-4ba4-95c0-4bfe41ae74df_calibrate_855820_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:01 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:02 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 556 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 01719a839fe8bb776e852148015f69c3 with result from task 556
parsl.dataflow.dflow DEBUG: Task 556 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 557 has memoization hash bceb78d5ecbee0b649c52fba08ff2eac
parsl.dataflow.memoization INFO: Task 557 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 557 try 0 launched on executor multi with executor id 1065
parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/8e6d41da-3052-492a-8b0d-179d8661465a_calibrate_855822_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/8e6d41da-3052-492a-8b0d-179d8661465a_calibrate_855822_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/1528f555-9382-4d5f-a6c3-7dc1ee12e688_characterizeImage_855822_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/1528f555-9382-4d5f-a6c3-7dc1ee12e688_characterizeImage_855822_35.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:02 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 394 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 47f6bc9c712806bcec4947f6c22e9aac with result from task 394
parsl.dataflow.dflow DEBUG: Task 394 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 395 has memoization hash 1467dbc045f207c00e13f6c07fac6124
parsl.dataflow.memoization INFO: Task 395 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 395 try 0 launched on executor multi with executor id 1066
parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/9e4eac21-5966-44bb-9207-a718fb4ccbcb_calibrate_855822_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/9e4eac21-5966-44bb-9207-a718fb4ccbcb_calibrate_855822_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/ae40b21c-156a-4a29-a615-7f5c6496433e_characterizeImage_855822_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/ae40b21c-156a-4a29-a615-7f5c6496433e_characterizeImage_855822_49.stderr
parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 346eb0a4d3474b48938c7a1eadffd1be with result from task 298
parsl.dataflow.dflow DEBUG: Task 298 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 299 has memoization hash 6a6cb6d14441031b82ef13c625fba3b2
parsl.dataflow.memoization INFO: Task 299 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:03 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 1067
parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/169c1aec-0fa4-489f-afc9-2f373222798c_calibrate_855821_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/169c1aec-0fa4-489f-afc9-2f373222798c_calibrate_855821_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ef28d0f7-f8a0-4f1d-bdb4-6d1dbbf188ad_characterizeImage_855821_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ef28d0f7-f8a0-4f1d-bdb4-6d1dbbf188ad_characterizeImage_855821_18.stderr
parsl.dataflow.dflow INFO: Task 967 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9b7a38a91c95c12090e05767346096f3 with result from task 967
parsl.dataflow.dflow DEBUG: Task 967 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 968 has memoization hash b5cabe1b4def485838cd814637e94707
parsl.dataflow.memoization INFO: Task 968 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 968 try 0 launched on executor multi with executor id 1068
parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/86e330e7-f7a8-423e-88ee-99ba88ed4dc0_calibrate_855822_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/86e330e7-f7a8-423e-88ee-99ba88ed4dc0_calibrate_855822_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/050feb2b-e2d5-4836-a789-83b143629ebb_characterizeImage_855822_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/050feb2b-e2d5-4836-a789-83b143629ebb_characterizeImage_855822_33.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:03 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 391 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2d0bdfc1afe841916ee778fc7cd84eff with result from task 391
parsl.dataflow.dflow DEBUG: Task 391 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 392 has memoization hash 88a5e12d16a8564b82a20c81155f28e4
parsl.dataflow.memoization INFO: Task 392 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 392 try 0 launched on executor multi with executor id 1069
parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/9ceeb967-300a-42ca-9b39-8ca2418d9888_calibrate_855822_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/9ceeb967-300a-42ca-9b39-8ca2418d9888_calibrate_855822_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/6bdfbd99-1c0b-4835-b6dd-ea609699059a_characterizeImage_855822_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/6bdfbd99-1c0b-4835-b6dd-ea609699059a_characterizeImage_855822_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:04 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:05 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:05 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:06 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 595 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0214a4bc1c9ca198f836085b853b28b1 with result from task 595
parsl.dataflow.dflow DEBUG: Task 595 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 596 has memoization hash be081424f9cc89ee5e5af34479b739a6
parsl.dataflow.memoization INFO: Task 596 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 596 try 0 launched on executor multi with executor id 1070
parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f3f6ec6f-8880-42e0-94b9-6cd4f74d930f_calibrate_855821_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f3f6ec6f-8880-42e0-94b9-6cd4f74d930f_calibrate_855821_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/b3c2fb9b-8755-4044-ac83-c5c1eb493a72_characterizeImage_855821_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/b3c2fb9b-8755-4044-ac83-c5c1eb493a72_characterizeImage_855821_11.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:07 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 797 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7850d53773bedaf7197c12a469a861f9 with result from task 797
parsl.dataflow.dflow DEBUG: Task 797 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/2f79bc85-e74b-4bf8-937d-1aa18b746d7e_calibrate_855820_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/2f79bc85-e74b-4bf8-937d-1aa18b746d7e_calibrate_855820_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:07 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 787 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 983920ad48c6e8322f61365585e9ccc4 with result from task 787
parsl.dataflow.dflow DEBUG: Task 787 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 788 has memoization hash 24b98d218d40c368816b98b2164234f4
parsl.dataflow.memoization INFO: Task 788 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 788 try 0 launched on executor multi with executor id 1071
parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/3538f04c-4456-4716-af87-d03144e58fe4_calibrate_855821_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/3538f04c-4456-4716-af87-d03144e58fe4_calibrate_855821_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/adf5c93a-f1f7-4c1f-a9c6-5cdc7acb34fa_characterizeImage_855821_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/adf5c93a-f1f7-4c1f-a9c6-5cdc7acb34fa_characterizeImage_855821_29.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:08 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:08 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 892 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8ae029a88de79497dba0fa54e6668f39 with result from task 892
parsl.dataflow.dflow DEBUG: Task 892 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 893 has memoization hash 3a5bddd8599c260610667af637439821
parsl.dataflow.memoization INFO: Task 893 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156eaca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 893 try 0 launched on executor multi with executor id 1072
parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/38362949-e20a-4cbd-970f-254f11acec28_calibrate_855821_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/38362949-e20a-4cbd-970f-254f11acec28_calibrate_855821_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ddf3813f-0eac-4f77-be55-98a088d2ddf0_characterizeImage_855821_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ddf3813f-0eac-4f77-be55-98a088d2ddf0_characterizeImage_855821_40.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:09 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1a1f297d54ec23a54877181af69ffd67 with result from task 119
parsl.dataflow.dflow DEBUG: Task 119 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/0da864ba-e1ff-4ff6-88fe-7563d4db6311_calibrate_855822_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/0da864ba-e1ff-4ff6-88fe-7563d4db6311_calibrate_855822_19.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:10 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:10 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 355004781cacdcb133301495d03d3877 with result from task 167
parsl.dataflow.dflow DEBUG: Task 167 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/3a79a315-d0ad-4906-a080-c0f3e957db7d_calibrate_855822_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/3a79a315-d0ad-4906-a080-c0f3e957db7d_calibrate_855822_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:11 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:11 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:12 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:12 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 925 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6fb2de3aca395efe2ff1b4a17db636af with result from task 925
parsl.dataflow.dflow DEBUG: Task 925 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 926 has memoization hash 1c2d548ab328574600383b2e2a72f186
parsl.dataflow.memoization INFO: Task 926 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15728f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 926 try 0 launched on executor multi with executor id 1073
parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/635a8d87-da35-4e85-883b-d6c08a880fe7_calibrate_855821_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/635a8d87-da35-4e85-883b-d6c08a880fe7_calibrate_855821_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/08967090-c681-4b0d-83ba-c80fa931a75c_characterizeImage_855821_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/08967090-c681-4b0d-83ba-c80fa931a75c_characterizeImage_855821_60.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:13 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 469 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0e4b7b4aaaa9d6fbafdb9c3006f3a068 with result from task 469
parsl.dataflow.dflow DEBUG: Task 469 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 470 has memoization hash 36f35a6a47e47baaa5a87c7a8b1f0511
parsl.dataflow.memoization INFO: Task 470 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 470 try 0 launched on executor multi with executor id 1074
parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/1f180673-5a99-4a17-b050-c597327ba6cd_calibrate_855822_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/1f180673-5a99-4a17-b050-c597327ba6cd_calibrate_855822_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/343947a3-4527-47ff-aa03-2352ef2e7258_characterizeImage_855822_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/343947a3-4527-47ff-aa03-2352ef2e7258_characterizeImage_855822_30.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:14 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:14 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 715 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d82d4163e31d875a77caa0b92c4ea184 with result from task 715
parsl.dataflow.dflow DEBUG: Task 715 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 716 has memoization hash 9a19c98cb7f149cc74d8188ad18f381f
parsl.dataflow.memoization INFO: Task 716 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1580b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 716 try 0 launched on executor multi with executor id 1075
parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/a3080a4c-9d41-42fc-a5da-6a3f13036d5d_calibrate_855821_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/a3080a4c-9d41-42fc-a5da-6a3f13036d5d_calibrate_855821_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/5cd4c424-ca65-479e-a5ad-c688f93d3f65_characterizeImage_855821_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/5cd4c424-ca65-479e-a5ad-c688f93d3f65_characterizeImage_855821_14.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:17 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:18 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:18 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:19 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:19 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a14b9c0bcfa4a65c9d8badbf0b357b4e with result from task 259
parsl.dataflow.dflow DEBUG: Task 259 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 260 has memoization hash abc2d2025d57bd18ce59b3d5f8ea1c08
parsl.dataflow.memoization INFO: Task 260 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 1076
parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/cac2efd4-4499-442b-b283-2e0d5911687a_calibrate_855821_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/cac2efd4-4499-442b-b283-2e0d5911687a_calibrate_855821_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/11397273-b5f0-4be1-a654-807721f28642_characterizeImage_855821_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/11397273-b5f0-4be1-a654-807721f28642_characterizeImage_855821_38.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:20 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 946 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 92dc1932e2f1d936dac08ede74616cee with result from task 946
parsl.dataflow.dflow DEBUG: Task 946 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 947 has memoization hash e09680d998563915a0ea51ca128833f4
parsl.dataflow.memoization INFO: Task 947 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 947 try 0 launched on executor multi with executor id 1077
parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/6641271d-7427-4544-9b47-2723096b4a7c_calibrate_855822_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/6641271d-7427-4544-9b47-2723096b4a7c_calibrate_855822_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/960a4f8f-bdcb-4ab7-b3f8-91d3f67a398e_characterizeImage_855822_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/960a4f8f-bdcb-4ab7-b3f8-91d3f67a398e_characterizeImage_855822_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:20 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:21 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:21 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 688 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bda2fecdb22c638f1f7552671986c399 with result from task 688
parsl.dataflow.dflow DEBUG: Task 688 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 689 has memoization hash aa09892e5a389a694f75da78f33c35c4
parsl.dataflow.memoization INFO: Task 689 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15809760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 689 try 0 launched on executor multi with executor id 1078
parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/fff2f9be-58ad-426a-b714-9b45a6d7bc2f_calibrate_855821_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/fff2f9be-58ad-426a-b714-9b45a6d7bc2f_calibrate_855821_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/4cc337c6-113a-473d-a40c-ad11dd05b56c_characterizeImage_855821_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/4cc337c6-113a-473d-a40c-ad11dd05b56c_characterizeImage_855821_41.stderr
parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1baaf62d3b81c9e0450b764c5f9e9450 with result from task 245
parsl.dataflow.dflow DEBUG: Task 245 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/5b64dde7-dd73-483e-aaa7-e387d9e0ea96_calibrate_855820_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/5b64dde7-dd73-483e-aaa7-e387d9e0ea96_calibrate_855820_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:22 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1063 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7b8b2b324509518edf6d86e092304f04 with result from task 1063
parsl.dataflow.dflow DEBUG: Task 1063 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1064 has memoization hash e6a1575ef6fefaac00d25e74ffdcf389
parsl.dataflow.memoization INFO: Task 1064 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1064 try 0 launched on executor multi with executor id 1079
parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/ad0fb3b8-a99d-4de4-a400-de2a99dc4410_calibrate_855821_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/ad0fb3b8-a99d-4de4-a400-de2a99dc4410_calibrate_855821_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/b597245e-4743-4e60-bea7-c3fd021ba5dd_characterizeImage_855821_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/b597245e-4743-4e60-bea7-c3fd021ba5dd_characterizeImage_855821_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:23 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:23 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4cbb3e3e9ab9f5306f6f725e3e892615 with result from task 142
parsl.dataflow.dflow DEBUG: Task 142 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 30d59ab26d75bfd41ce09b034fa73091
parsl.dataflow.memoization INFO: Task 143 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a41c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 1080
parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/229d98ce-1c3b-481b-a949-8e69e552806d_calibrate_855821_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/229d98ce-1c3b-481b-a949-8e69e552806d_calibrate_855821_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/db6bc5f1-fc1e-4c4f-9dc8-b2f21acf45f6_characterizeImage_855821_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/db6bc5f1-fc1e-4c4f-9dc8-b2f21acf45f6_characterizeImage_855821_30.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:24 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1054 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8be1e0b43e97c14935cd527cae6cd5a9 with result from task 1054
parsl.dataflow.dflow DEBUG: Task 1054 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1055 has memoization hash 4be219b04d5e7c680ef1cae86ba5f041
parsl.dataflow.memoization INFO: Task 1055 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1055 try 0 launched on executor multi with executor id 1081
parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/601e9b63-afa3-4748-8b0f-e1f65a4032ed_calibrate_855821_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/601e9b63-afa3-4748-8b0f-e1f65a4032ed_calibrate_855821_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/beef6808-7a49-4e89-9015-95b28b6d89c4_characterizeImage_855821_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/beef6808-7a49-4e89-9015-95b28b6d89c4_characterizeImage_855821_1.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:24 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:25 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 934 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 17cec5ab4d994ed2c22832298dc57f52 with result from task 934
parsl.dataflow.dflow DEBUG: Task 934 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 935 has memoization hash 196c6d89c3943c7ee5a8eb227d1d9219
parsl.dataflow.memoization INFO: Task 935 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157298a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 935 try 0 launched on executor multi with executor id 1082
parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/4a9e6ad6-c108-4237-8382-f4a7042d73ac_calibrate_855821_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/4a9e6ad6-c108-4237-8382-f4a7042d73ac_calibrate_855821_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/14b2f87d-08ad-4d7b-a078-a8301b1539e3_characterizeImage_855821_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/14b2f87d-08ad-4d7b-a078-a8301b1539e3_characterizeImage_855821_4.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:26 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 379 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e14e5fa0c5b18b79c04e7d599602fdec with result from task 379
parsl.dataflow.dflow DEBUG: Task 379 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 380 has memoization hash 684e170fd0c2b7c1a6b24b7593a6b83b
parsl.dataflow.memoization INFO: Task 380 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158e1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 380 try 0 launched on executor multi with executor id 1083
parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/576cf0de-e999-49a6-83a7-23d1ae8d5d3b_calibrate_855821_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/576cf0de-e999-49a6-83a7-23d1ae8d5d3b_calibrate_855821_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/6ac9e124-f654-4a89-882a-46f793ae9dd8_characterizeImage_855821_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/6ac9e124-f654-4a89-882a-46f793ae9dd8_characterizeImage_855821_48.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:26 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:27 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 457 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 06544cafc52b6bc8433f4458aa751c7e with result from task 457
parsl.dataflow.dflow DEBUG: Task 457 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 458 has memoization hash d03efc6c696cddd687c9f7039efa752e
parsl.dataflow.memoization INFO: Task 458 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 458 try 0 launched on executor multi with executor id 1084
parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/30cecf1c-51d3-4834-8c5a-0a79a6bc59b2_calibrate_855822_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/30cecf1c-51d3-4834-8c5a-0a79a6bc59b2_calibrate_855822_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/98ffe426-8edd-4eea-92b8-392dd8c1543d_characterizeImage_855822_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/98ffe426-8edd-4eea-92b8-392dd8c1543d_characterizeImage_855822_36.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:28 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 902 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 37f04b408c7fc1d5e5f144d83074386b with result from task 902
parsl.dataflow.dflow DEBUG: Task 902 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/cca711f2-1e49-4a53-a028-0a7027af994a_calibrate_855821_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/cca711f2-1e49-4a53-a028-0a7027af994a_calibrate_855821_39.stderr
parsl.dataflow.dflow INFO: Task 1069 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bef6b320f73b3af41886fac2281572ff with result from task 1069
parsl.dataflow.dflow DEBUG: Task 1069 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1070 has memoization hash 1ce26dda1eab16199778920b16b51d04
parsl.dataflow.memoization INFO: Task 1070 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1070 try 0 launched on executor multi with executor id 1085
parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/a92bcac1-0e54-4470-882c-a0d22a2d54c7_calibrate_855821_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/a92bcac1-0e54-4470-882c-a0d22a2d54c7_calibrate_855821_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ef62fb4f-64d0-4174-bac3-4504860186c9_characterizeImage_855821_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ef62fb4f-64d0-4174-bac3-4504860186c9_characterizeImage_855821_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:28 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1031 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ecad41b48309fba277acd916cc2cd3ac with result from task 1031
parsl.dataflow.dflow DEBUG: Task 1031 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/7fb9c1b8-6a04-4ced-ab27-b5f12e664ff8_calibrate_855821_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/7fb9c1b8-6a04-4ced-ab27-b5f12e664ff8_calibrate_855821_19.stderr
parsl.dataflow.dflow INFO: Task 919 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9a6a16168c3637605d65882f5bcbf9c8 with result from task 919
parsl.dataflow.dflow DEBUG: Task 919 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 920 has memoization hash 9cdbd77438a63bc09de5cff152ac516c
parsl.dataflow.memoization INFO: Task 920 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15728900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 920 try 0 launched on executor multi with executor id 1086
parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/f41daa24-9816-4b10-aa1e-fce85b17b6fc_calibrate_855822_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/f41daa24-9816-4b10-aa1e-fce85b17b6fc_calibrate_855822_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/e1237aea-99a6-479d-8eb6-9bde68f1a5b1_characterizeImage_855822_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/e1237aea-99a6-479d-8eb6-9bde68f1a5b1_characterizeImage_855822_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 120
parsl.dataflow.strategy DEBUG: Executor multi has 120 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:00:29 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:00:29 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:29 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:30 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:31 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:31 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 784 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3e4782217010717f700cd3bd61343a43 with result from task 784
parsl.dataflow.dflow DEBUG: Task 784 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 785 has memoization hash 0d66338e7423208dbb627b6731b2fdd3
parsl.dataflow.memoization INFO: Task 785 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 785 try 0 launched on executor multi with executor id 1087
parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/6f5464f8-3625-4389-bbc5-8ad76b83ff3a_calibrate_855821_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/6f5464f8-3625-4389-bbc5-8ad76b83ff3a_calibrate_855821_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/dd7f1490-10b2-44c5-a417-10c061b85a61_characterizeImage_855821_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/dd7f1490-10b2-44c5-a417-10c061b85a61_characterizeImage_855821_23.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:32 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 949 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f42aecc1cfb330182a20374ddf1b7df6 with result from task 949
parsl.dataflow.dflow DEBUG: Task 949 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 950 has memoization hash b42d33be99e5d0f9a1ffb5ab8ed717d3
parsl.dataflow.memoization INFO: Task 950 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1572a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 950 try 0 launched on executor multi with executor id 1088
parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/e330ef32-49e0-413a-b093-7de9377a85ad_calibrate_855821_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/e330ef32-49e0-413a-b093-7de9377a85ad_calibrate_855821_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/c28f3221-e2f2-456c-9ceb-7b405ff67665_characterizeImage_855821_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/c28f3221-e2f2-456c-9ceb-7b405ff67665_characterizeImage_855821_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:33 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:33 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 427 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6ea846b3acb4147e72347a118b3f47c6 with result from task 427
parsl.dataflow.dflow DEBUG: Task 427 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 428 has memoization hash 58d17b27ec666faa36ef43c9d4ceb03a
parsl.dataflow.memoization INFO: Task 428 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 1089
parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/44f6712d-4ea8-4075-9450-4a22f2d0b55b_calibrate_855821_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/44f6712d-4ea8-4075-9450-4a22f2d0b55b_calibrate_855821_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/7e91f3bc-72b0-4250-a184-f5c9225e1839_characterizeImage_855821_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/7e91f3bc-72b0-4250-a184-f5c9225e1839_characterizeImage_855821_54.stderr
parsl.dataflow.dflow INFO: Task 644 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f10b63ced8231be7f95ae50b64abd647 with result from task 644
parsl.dataflow.dflow DEBUG: Task 644 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/b18b9a5f-0928-40b4-a475-53694ab297f3_calibrate_855822_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/b18b9a5f-0928-40b4-a475-53694ab297f3_calibrate_855822_45.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:34 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a754a308a897ac88bcf88cfd7690d033 with result from task 85
parsl.dataflow.dflow DEBUG: Task 85 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 37af65d83481ab69e24e9f860ce1449f
parsl.dataflow.memoization INFO: Task 86 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 1090
parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/6c9ceaaa-b547-4c46-ac57-c8628c8b7c03_calibrate_855821_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/6c9ceaaa-b547-4c46-ac57-c8628c8b7c03_calibrate_855821_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/e3fa9959-4fc0-43f8-8bca-2cb22c80a080_characterizeImage_855821_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/e3fa9959-4fc0-43f8-8bca-2cb22c80a080_characterizeImage_855821_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:34 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:35 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:35 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:36 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:37 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:37 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 622 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b0d4c0f24bae188e7ff352428ec7f5fa with result from task 622
parsl.dataflow.dflow DEBUG: Task 622 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 623 has memoization hash a99d3eb691470226b12553fb1f834c32
parsl.dataflow.memoization INFO: Task 623 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 623 try 0 launched on executor multi with executor id 1091
parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/74cd66d8-5bc4-45eb-8541-34da01549a0d_calibrate_855821_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/74cd66d8-5bc4-45eb-8541-34da01549a0d_calibrate_855821_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ef21c387-1bbf-4c6b-8687-c3d6db89bbb1_characterizeImage_855821_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/ef21c387-1bbf-4c6b-8687-c3d6db89bbb1_characterizeImage_855821_57.stderr
parsl.dataflow.dflow INFO: Task 997 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7e7bb74245950934264139f22d6e6170 with result from task 997
parsl.dataflow.dflow DEBUG: Task 997 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 998 has memoization hash 8c74f894f290e288a4329cdaa91fd51f
parsl.dataflow.memoization INFO: Task 998 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 998 try 0 launched on executor multi with executor id 1092
parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/81bbb656-c2a0-4451-a1c5-6f8098500370_calibrate_855822_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/81bbb656-c2a0-4451-a1c5-6f8098500370_calibrate_855822_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/86659ec4-e2f0-47af-8918-e5bfc2fc579f_characterizeImage_855822_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/86659ec4-e2f0-47af-8918-e5bfc2fc579f_characterizeImage_855822_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:38 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:38 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 439 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7c881bb88c4ac7559a8802d20c12978c with result from task 439
parsl.dataflow.dflow DEBUG: Task 439 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 440 has memoization hash 6693a32c20e7d218a67fc07211bc6ac4
parsl.dataflow.memoization INFO: Task 440 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 440 try 0 launched on executor multi with executor id 1093
parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/e75793fb-c2b4-445e-8426-271abcce49d5_calibrate_855821_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/e75793fb-c2b4-445e-8426-271abcce49d5_calibrate_855821_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/b2845ea8-710e-49b7-88fb-9ff22b16b5ae_characterizeImage_855821_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/b2845ea8-710e-49b7-88fb-9ff22b16b5ae_characterizeImage_855821_24.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:39 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 87df33c52d32e2506dd9f74cd30748d7 with result from task 317
parsl.dataflow.dflow DEBUG: Task 317 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/c6442850-468a-4c9c-94a9-e7b526acf47c_calibrate_855822_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/c6442850-468a-4c9c-94a9-e7b526acf47c_calibrate_855822_1.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:40 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 64d3abfd5a280f72b53916f973f500e4 with result from task 79
parsl.dataflow.dflow DEBUG: Task 79 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 80 has memoization hash c823ddd9abc13538e03d5d7df93aed6e
parsl.dataflow.memoization INFO: Task 80 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 1094
parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f82adf30-0d72-4925-a686-7040cdc694ee_calibrate_855821_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f82adf30-0d72-4925-a686-7040cdc694ee_calibrate_855821_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/41865dfc-30a2-44dd-ac08-23a65c066bbf_characterizeImage_855821_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/41865dfc-30a2-44dd-ac08-23a65c066bbf_characterizeImage_855821_16.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:40 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:41 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:41 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:42 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3870db251893260bd462350f205843ee with result from task 221
parsl.dataflow.dflow DEBUG: Task 221 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/3fd83976-c949-48d3-a9e6-af1ecfb422a9_calibrate_855822_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/3fd83976-c949-48d3-a9e6-af1ecfb422a9_calibrate_855822_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:42 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:43 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 491 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c2d3b18b31485bdb458273ec74804a75 with result from task 491
parsl.dataflow.dflow DEBUG: Task 491 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/ff879527-4f7f-4400-b240-2843b38b4517_calibrate_855821_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/ff879527-4f7f-4400-b240-2843b38b4517_calibrate_855821_3.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:44 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:44 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8cf0336645e9b9d64523a7d198234f2a with result from task 26
parsl.dataflow.dflow DEBUG: Task 26 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/22cdaa20-484f-4cf6-b076-19598f61ed07_calibrate_855822_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/22cdaa20-484f-4cf6-b076-19598f61ed07_calibrate_855822_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:45 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:45 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e0d2cf545999cae267e7485b97bbcc7e with result from task 353
parsl.dataflow.dflow DEBUG: Task 353 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/16142cd8-ad3d-4f37-869d-3dbc0627c553_calibrate_855821_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/16142cd8-ad3d-4f37-869d-3dbc0627c553_calibrate_855821_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:46 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:46 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:47 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:48 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.dataflow.dflow INFO: Task 1009 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b0715db9d5e46d9fa36de467df3d1638 with result from task 1009
parsl.dataflow.dflow DEBUG: Task 1009 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1010 has memoization hash ba3fdcb62de7b3c6c06c47bd61ff9c22
parsl.dataflow.memoization INFO: Task 1010 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1010 try 0 launched on executor multi with executor id 1095
parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/b2fc135c-f1d6-405d-9db6-215b12f87138_calibrate_855822_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/b2fc135c-f1d6-405d-9db6-215b12f87138_calibrate_855822_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a0faff6d-ecca-4486-82d5-3c0f1176aa05_characterizeImage_855822_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a0faff6d-ecca-4486-82d5-3c0f1176aa05_characterizeImage_855822_29.stderr
parsl.dataflow.dflow INFO: Task 565 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d076223a307a029bebf539c147350d77 with result from task 565
parsl.dataflow.dflow DEBUG: Task 565 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 566 has memoization hash da2d8d69acd7d836afd74c7c95e89cd7
parsl.dataflow.memoization INFO: Task 566 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 566 try 0 launched on executor multi with executor id 1096
parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/3d2f7ae9-0347-45ba-9c72-5fd5afa5b102_calibrate_855822_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/3d2f7ae9-0347-45ba-9c72-5fd5afa5b102_calibrate_855822_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/9f7493a5-1ca2-4541-b84d-b795883c4ed8_characterizeImage_855822_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/9f7493a5-1ca2-4541-b84d-b795883c4ed8_characterizeImage_855822_53.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:50 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 818870c1c768f0b93651cb178764d8f6 with result from task 251
parsl.dataflow.dflow DEBUG: Task 251 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/32fb4016-a4cc-4036-8202-576add071d07_calibrate_855822_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/32fb4016-a4cc-4036-8202-576add071d07_calibrate_855822_15.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:50 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d44d1ad2dc0741a55b0dadf23230749e with result from task 286
parsl.dataflow.dflow DEBUG: Task 286 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 75d3851465932ffe5f7bd5ea714f15a3
parsl.dataflow.memoization INFO: Task 287 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a8f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 1097
parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/0d195381-7b2f-4aca-ad71-15cd43b22136_calibrate_855822_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/0d195381-7b2f-4aca-ad71-15cd43b22136_calibrate_855822_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/154738c4-bdef-4810-87ee-747edb340511_characterizeImage_855822_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/154738c4-bdef-4810-87ee-747edb340511_characterizeImage_855822_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:51 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:52 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7b44b8d8d954dd25b90ce1c4ba93fe22 with result from task 175
parsl.dataflow.dflow DEBUG: Task 175 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 176 has memoization hash cfc2c80c64397f650268d8077cf2f4d2
parsl.dataflow.memoization INFO: Task 176 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a43e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 1098
parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/3fb765e2-afc2-498d-b1e1-f0ed2f76c443_calibrate_855822_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/3fb765e2-afc2-498d-b1e1-f0ed2f76c443_calibrate_855822_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/f50baa40-956a-4cea-9495-2f8bb14c5c52_characterizeImage_855822_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/f50baa40-956a-4cea-9495-2f8bb14c5c52_characterizeImage_855822_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:52 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 448969340e2e9d1787291ab06f674e46 with result from task 236
parsl.dataflow.dflow DEBUG: Task 236 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/38cb4ed5-5cfa-4d34-b98c-e9355f9bc5a8_calibrate_855821_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/38cb4ed5-5cfa-4d34-b98c-e9355f9bc5a8_calibrate_855821_31.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:53 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:53 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 728 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6fecf814449aaef1537b3c17187f924d with result from task 728
parsl.dataflow.dflow DEBUG: Task 728 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/bcc5b1c7-e62a-4f08-b00c-f8bf24bb99ce_calibrate_855822_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/bcc5b1c7-e62a-4f08-b00c-f8bf24bb99ce_calibrate_855822_31.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:54 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:55 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:55 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 829 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f0b50c199d967c0f4078ca0e38af84ee with result from task 829
parsl.dataflow.dflow DEBUG: Task 829 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 830 has memoization hash 328de90be88149aa10552060e9132a0f
parsl.dataflow.memoization INFO: Task 830 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15896ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 830 try 0 launched on executor multi with executor id 1099
parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/934539a4-0a94-4872-b61d-13ce098d5217_calibrate_855821_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/934539a4-0a94-4872-b61d-13ce098d5217_calibrate_855821_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/baee5428-1d8a-4963-8c53-51c5e8aaf499_characterizeImage_855821_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/baee5428-1d8a-4963-8c53-51c5e8aaf499_characterizeImage_855821_47.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:56 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:57 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 754 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e731a5119abc57c7bb562192418fab12 with result from task 754
parsl.dataflow.dflow DEBUG: Task 754 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 755 has memoization hash 4ec8d1ee20092d238b933d7c842037f3
parsl.dataflow.memoization INFO: Task 755 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15859c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 755 try 0 launched on executor multi with executor id 1100
parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/0a98e223-df1b-45e2-961e-e5ba45bd7f87_calibrate_855822_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/0a98e223-df1b-45e2-961e-e5ba45bd7f87_calibrate_855822_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/7796db12-2c74-47de-ab9b-64c7763d13a9_characterizeImage_855822_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/7796db12-2c74-47de-ab9b-64c7763d13a9_characterizeImage_855822_47.stderr
parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 09f402290d993f848af5644b97bee408 with result from task 58
parsl.dataflow.dflow DEBUG: Task 58 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 59 has memoization hash da11ac0ef3a8a5643410f50f0051dc21
parsl.dataflow.memoization INFO: Task 59 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a1c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 1101
parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/aef71b02-5c43-48b7-868a-6e1ea31c3d0e_calibrate_855822_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/aef71b02-5c43-48b7-868a-6e1ea31c3d0e_calibrate_855822_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/6bee672c-7f28-438b-a7b2-fd625a1c2f5b_characterizeImage_855822_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/6bee672c-7f28-438b-a7b2-fd625a1c2f5b_characterizeImage_855822_41.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:57 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:58 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 872 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 869c6921f656b3eb185315f8cd2e26c1 with result from task 872
parsl.dataflow.dflow DEBUG: Task 872 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/021c6d30-7156-4ae0-a9a7-0063601c10f5_calibrate_855822_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/021c6d30-7156-4ae0-a9a7-0063601c10f5_calibrate_855822_32.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:58 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 1060 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7f894eae81e630b563b8c12ba27b025f with result from task 1060
parsl.dataflow.dflow DEBUG: Task 1060 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1061 has memoization hash 330eb21d561957be0b65d9b1cfa006fb
parsl.dataflow.memoization INFO: Task 1061 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1061 try 0 launched on executor multi with executor id 1102
parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/3e6c36ee-3974-4e47-9392-8a38cbed1d04_calibrate_855822_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/3e6c36ee-3974-4e47-9392-8a38cbed1d04_calibrate_855822_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/8d62c655-4f6c-4376-bcf8-167e5d25b8f0_characterizeImage_855822_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/8d62c655-4f6c-4376-bcf8-167e5d25b8f0_characterizeImage_855822_38.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:00:59 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:00 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:00 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:01 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:01 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:02 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:03 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:03 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 811 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cf75a9fc448acef23c62ece03e93effe with result from task 811
parsl.dataflow.dflow DEBUG: Task 811 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 812 has memoization hash 4470d2697b03849686cf425edd769fb3
parsl.dataflow.memoization INFO: Task 812 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15895800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 812 try 0 launched on executor multi with executor id 1103
parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/28342909-b282-4234-8b8f-c2280049d84b_calibrate_855822_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/28342909-b282-4234-8b8f-c2280049d84b_calibrate_855822_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/7ce92523-1adf-49ac-bebc-1a6d3e1431d7_characterizeImage_855822_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/7ce92523-1adf-49ac-bebc-1a6d3e1431d7_characterizeImage_855822_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:04 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:04 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d9797ff3b14cce5933db53d1df8fc5de with result from task 302
parsl.dataflow.dflow DEBUG: Task 302 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f84fb7a1-7df0-433e-a39c-47a874fac8ab_calibrate_855821_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f84fb7a1-7df0-433e-a39c-47a874fac8ab_calibrate_855821_15.stderr
parsl.dataflow.dflow INFO: Task 991 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4b2946970080e9d2108da395e5075440 with result from task 991
parsl.dataflow.dflow DEBUG: Task 991 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 992 has memoization hash dd66279ac0af0e750b13a396df39ef30
parsl.dataflow.memoization INFO: Task 992 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1577d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 992 try 0 launched on executor multi with executor id 1104
parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/96610ef8-7691-4720-a081-102a0a92353c_calibrate_855822_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/96610ef8-7691-4720-a081-102a0a92353c_calibrate_855822_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a36ab353-27f1-45b0-a927-5f2adda996cf_characterizeImage_855822_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/a36ab353-27f1-45b0-a927-5f2adda996cf_characterizeImage_855822_34.stderr
parsl.dataflow.dflow INFO: Task 446 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cecc914c1a08955ac1def65633cb8877 with result from task 446
parsl.dataflow.dflow DEBUG: Task 446 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/5682b7f5-433f-422c-9e61-f6cab04b2dd0_calibrate_855821_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/5682b7f5-433f-422c-9e61-f6cab04b2dd0_calibrate_855821_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:05 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 425 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 651d2671989b7f6dadc4b9a64a248d8f with result from task 425
parsl.dataflow.dflow DEBUG: Task 425 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/8dc435b4-69bd-413c-afaf-688fa44e4282_calibrate_855821_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/8dc435b4-69bd-413c-afaf-688fa44e4282_calibrate_855821_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:06 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:06 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:07 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:07 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:08 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:09 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 405cf414ab96d42e2b15581da218aeb9 with result from task 341
parsl.dataflow.dflow DEBUG: Task 341 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/b7d6d7a1-5bc6-442c-a65e-33a807b51bea_calibrate_855821_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/b7d6d7a1-5bc6-442c-a65e-33a807b51bea_calibrate_855821_13.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:09 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:10 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:10 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 841 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5c4bba1302ef44cb51e4638acc8d3ffe with result from task 841
parsl.dataflow.dflow DEBUG: Task 841 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 842 has memoization hash 3e9a31038eab6155385cc5daffba15b6
parsl.dataflow.memoization INFO: Task 842 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15897740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:11 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Parsl task 842 try 0 launched on executor multi with executor id 1105
parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/0cb1aaa3-c51d-4f49-b76c-3a3ca8c8760c_calibrate_855821_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/0cb1aaa3-c51d-4f49-b76c-3a3ca8c8760c_calibrate_855821_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/afe3182f-93d7-45a7-8779-80cee2563e76_characterizeImage_855821_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/afe3182f-93d7-45a7-8779-80cee2563e76_characterizeImage_855821_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:12 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:12 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:13 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:13 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 974 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 369d56df2939e4e236df3c176fea0af3 with result from task 974
parsl.dataflow.dflow DEBUG: Task 974 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f250d9ed-ab5c-406b-9278-17f05c630f8c_calibrate_855821_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f250d9ed-ab5c-406b-9278-17f05c630f8c_calibrate_855821_5.stderr
parsl.dataflow.dflow INFO: Task 937 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 86156130ad2a683eee2e9487875854d4 with result from task 937
parsl.dataflow.dflow DEBUG: Task 937 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 938 has memoization hash ac6ff164c371e2b243a2527e8b4deb36
parsl.dataflow.memoization INFO: Task 938 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15729bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 938 try 0 launched on executor multi with executor id 1106
parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/ff1f6ee0-4186-421f-80a4-e60ecbc8b6da_calibrate_855822_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/ff1f6ee0-4186-421f-80a4-e60ecbc8b6da_calibrate_855822_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/23dafb7e-ace3-40b5-bbcb-60cbd204de5f_characterizeImage_855822_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/23dafb7e-ace3-40b5-bbcb-60cbd204de5f_characterizeImage_855822_24.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:14 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:15 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a93c3a47d61ca93ffb05f3875603c976 with result from task 184
parsl.dataflow.dflow DEBUG: Task 184 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 185 has memoization hash f6486daf4e50b1d6e94aa48316d9f520
parsl.dataflow.memoization INFO: Task 185 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a64860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 1107
parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f3d7bb04-4231-4877-b038-1430451fbeca_calibrate_855821_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f3d7bb04-4231-4877-b038-1430451fbeca_calibrate_855821_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/bb148c1a-6ace-4fd1-a3ae-0a96242c60c5_characterizeImage_855821_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/bb148c1a-6ace-4fd1-a3ae-0a96242c60c5_characterizeImage_855821_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:15 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:16 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:16 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a9bce1c8c219e8301551e107d116b838 with result from task 227
parsl.dataflow.dflow DEBUG: Task 227 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/53cd8091-bd69-41d3-96bb-7bc78b9e3bdd_calibrate_855821_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/53cd8091-bd69-41d3-96bb-7bc78b9e3bdd_calibrate_855821_33.stderr
parsl.dataflow.dflow INFO: Task 691 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ed7a63e97f2608ca47b2bfec1d7f3687 with result from task 691
parsl.dataflow.dflow DEBUG: Task 691 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 692 has memoization hash fd08c71c7e8df1f193920e22ba645dbd
parsl.dataflow.memoization INFO: Task 692 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15809a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 692 try 0 launched on executor multi with executor id 1108
parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/6148ed34-f8b3-4751-b8a2-f17263f52d12_calibrate_855822_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/6148ed34-f8b3-4751-b8a2-f17263f52d12_calibrate_855822_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/73bb3146-8820-49c9-abff-b3f5384f1dcf_characterizeImage_855822_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/73bb3146-8820-49c9-abff-b3f5384f1dcf_characterizeImage_855822_50.stderr
parsl.dataflow.dflow INFO: Task 1048 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d7a41152fa38e92ef0eed37d5f8443c8 with result from task 1048
parsl.dataflow.dflow DEBUG: Task 1048 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1049 has memoization hash f229545f88c40fc7e7400f0bbfc2f1d6
parsl.dataflow.memoization INFO: Task 1049 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1049 try 0 launched on executor multi with executor id 1109
parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/59cb3f1d-4616-45fd-be15-eda988d3a794_calibrate_855821_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/59cb3f1d-4616-45fd-be15-eda988d3a794_calibrate_855821_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/f2385b88-a091-40b0-a28f-63d340c9ad4c_characterizeImage_855821_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/f2385b88-a091-40b0-a28f-63d340c9ad4c_characterizeImage_855821_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:17 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f9fe5af4bcaf595da6218b5b08309f04 with result from task 230
parsl.dataflow.dflow DEBUG: Task 230 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/a48b7ec2-3997-4c09-b646-5ed53f367e59_calibrate_855822_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/a48b7ec2-3997-4c09-b646-5ed53f367e59_calibrate_855822_39.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:18 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:18 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 545 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cc75cec9ed896257b833f540fbf4ed9b with result from task 545
parsl.dataflow.dflow DEBUG: Task 545 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/a6766b0b-56c6-40c5-bd05-76f2da1960bc_calibrate_855822_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/a6766b0b-56c6-40c5-bd05-76f2da1960bc_calibrate_855822_3.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:19 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9c0ebad85b276f5d9bfe9f05fa9f969d with result from task 116
parsl.dataflow.dflow DEBUG: Task 116 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/bb716244-b9c3-4a8c-86de-28ba123f9c5c_calibrate_855821_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/bb716244-b9c3-4a8c-86de-28ba123f9c5c_calibrate_855821_32.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:19 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 604 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0dd5d1269c52769f4ab68dd7586ab31c with result from task 604
parsl.dataflow.dflow DEBUG: Task 604 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 605 has memoization hash e2d0af72597aec457e0a9beff42f3177
parsl.dataflow.memoization INFO: Task 605 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 605 try 0 launched on executor multi with executor id 1110
parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/aaa5c5d4-71ab-4638-a7ea-a78f488da735_calibrate_855822_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/aaa5c5d4-71ab-4638-a7ea-a78f488da735_calibrate_855822_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/70ee040c-fc43-43b5-8d66-1315f5be4080_characterizeImage_855822_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/70ee040c-fc43-43b5-8d66-1315f5be4080_characterizeImage_855822_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:20 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 701 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f668af1d02503b2f85e075c994e4eee5 with result from task 701
parsl.dataflow.dflow DEBUG: Task 701 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/fcbc7466-e3a2-4afa-944f-76264fdc060b_calibrate_855821_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/fcbc7466-e3a2-4afa-944f-76264fdc060b_calibrate_855821_45.stderr
parsl.dataflow.dflow INFO: Task 862 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ab2e436003bc574cc0c85e4995dda5c2 with result from task 862
parsl.dataflow.dflow DEBUG: Task 862 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 863 has memoization hash ecf69be460945284dbbc413fdc7f969c
parsl.dataflow.memoization INFO: Task 863 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156e8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 863 try 0 launched on executor multi with executor id 1111
parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/2ab6a07b-f047-4619-bc76-4d64d6d07af4_calibrate_855821_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/2ab6a07b-f047-4619-bc76-4d64d6d07af4_calibrate_855821_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/268f5621-29f5-4259-963d-55edc671e1df_characterizeImage_855821_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/268f5621-29f5-4259-963d-55edc671e1df_characterizeImage_855821_53.stderr
parsl.dataflow.dflow INFO: Task 401 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f7f4b16239d05e46b3ca634a91fa0463 with result from task 401
parsl.dataflow.dflow DEBUG: Task 401 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/667c25f4-5fe7-4dcd-8e60-7ac5f417fc47_calibrate_855822_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/667c25f4-5fe7-4dcd-8e60-7ac5f417fc47_calibrate_855822_40.stderr
parsl.dataflow.dflow INFO: Task 1034 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 270b3a47e0d167936b28350e67a2899c with result from task 1034
parsl.dataflow.dflow DEBUG: Task 1034 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/918ca5d8-9d7c-4700-804c-b4d3f1ac3498_calibrate_855822_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/918ca5d8-9d7c-4700-804c-b4d3f1ac3498_calibrate_855822_4.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:23 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1115 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 70cb3710309791d2e01d705bd8099a4b with result from task 1115
parsl.dataflow.dflow DEBUG: Task 1115 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/1c47251c-2f68-446c-8341-444c6c8b4add_calibrate_855822_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/1c47251c-2f68-446c-8341-444c6c8b4add_calibrate_855822_14.stderr
parsl.dataflow.dflow INFO: Task 562 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ad86e64ad4fed28b9d9c383cbd8b9d4b with result from task 562
parsl.dataflow.dflow DEBUG: Task 562 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 563 has memoization hash abe37a008c96542fdf956973e800fa21
parsl.dataflow.memoization INFO: Task 563 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 563 try 0 launched on executor multi with executor id 1112
parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/ef8d9ca7-1c70-4ab6-89f7-25536e7f5879_calibrate_855821_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/ef8d9ca7-1c70-4ab6-89f7-25536e7f5879_calibrate_855821_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/87a40593-9647-4c1e-88d8-f549b330a5b5_characterizeImage_855821_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/87a40593-9647-4c1e-88d8-f549b330a5b5_characterizeImage_855821_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:23 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:24 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:24 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 769 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8c860e386ac2cd8013616714e50e0dd6 with result from task 769
parsl.dataflow.dflow DEBUG: Task 769 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 770 has memoization hash f98f1df3cedf7de1054aca98006e412b
parsl.dataflow.memoization INFO: Task 770 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1585ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 770 try 0 launched on executor multi with executor id 1113
parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/ee50eb3d-7f44-475b-9e12-fc0b468bb6df_calibrate_855821_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/ee50eb3d-7f44-475b-9e12-fc0b468bb6df_calibrate_855821_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/c3b10488-92b9-4c2f-bac8-a7082a9178d2_characterizeImage_855821_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/c3b10488-92b9-4c2f-bac8-a7082a9178d2_characterizeImage_855821_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:25 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 082b65c7a03299ea4bfed63004b61177 with result from task 52
parsl.dataflow.dflow DEBUG: Task 52 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 53 has memoization hash e5f9162cef2077b471ab68226fc895b8
parsl.dataflow.memoization INFO: Task 53 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa169fff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 1114
parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/f4a1b476-e7b7-49a9-b165-47c0564a0733_calibrate_855822_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/f4a1b476-e7b7-49a9-b165-47c0564a0733_calibrate_855822_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/7fed83ac-fb51-430a-b554-a0d20f137966_characterizeImage_855822_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/7fed83ac-fb51-430a-b554-a0d20f137966_characterizeImage_855822_48.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:26 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry de465bd65ddf630660d67c427b14aa2e with result from task 32
parsl.dataflow.dflow DEBUG: Task 32 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/a8fc75fe-554a-4aab-9ab5-3da9ee82e81c_calibrate_855822_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/a8fc75fe-554a-4aab-9ab5-3da9ee82e81c_calibrate_855822_8.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:26 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 815 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b9b87266e39c87574dc77bb4180072be with result from task 815
parsl.dataflow.dflow DEBUG: Task 815 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/6fa3f0b6-7a83-4ac8-b9ee-41cd0a3f0a03_calibrate_855821_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/6fa3f0b6-7a83-4ac8-b9ee-41cd0a3f0a03_calibrate_855821_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:27 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 95
parsl.dataflow.strategy DEBUG: Executor multi has 95 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:01:28 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:01:28 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:28 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:29 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 80b77ac3b3dc963de57d1f3de2118521 with result from task 20
parsl.dataflow.dflow DEBUG: Task 20 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/d93ff072-bd90-4756-95ba-28ee1544ccee_calibrate_855822_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/d93ff072-bd90-4756-95ba-28ee1544ccee_calibrate_855822_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:30 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:30 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:31 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 18421befe5b73c0ed1a797a04499c87f with result from task 196
parsl.dataflow.dflow DEBUG: Task 196 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 197 has memoization hash fa0c4b0e63f026ffea1225d6253fff30
parsl.dataflow.memoization INFO: Task 197 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a654e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 1115
parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/b03016eb-66dd-4d6a-bae7-16cce60d9d3a_calibrate_855822_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/b03016eb-66dd-4d6a-bae7-16cce60d9d3a_calibrate_855822_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/2ef67a39-112c-4dad-b8df-1174db3d4512_characterizeImage_855822_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/2ef67a39-112c-4dad-b8df-1174db3d4512_characterizeImage_855822_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:31 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:32 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:33 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:33 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:34 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:35 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:35 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 404 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3b06e1557724b82d051a141c761a3577 with result from task 404
parsl.dataflow.dflow DEBUG: Task 404 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/390a908d-86cd-4dca-8be9-7667af82a98e_calibrate_855821_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/390a908d-86cd-4dca-8be9-7667af82a98e_calibrate_855821_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:36 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 749 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7c11738386018701249836cb58b1f535 with result from task 749
parsl.dataflow.dflow DEBUG: Task 749 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/d1e992de-1295-4fac-ae36-fe7149cc179e_calibrate_855821_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/d1e992de-1295-4fac-ae36-fe7149cc179e_calibrate_855821_22.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:36 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 599 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4313f5b88be1190a8b240aa65e16dccf with result from task 599
parsl.dataflow.dflow DEBUG: Task 599 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/9cc208aa-ecdf-44c3-9e57-ecf0e5f4896b_calibrate_855821_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/9cc208aa-ecdf-44c3-9e57-ecf0e5f4896b_calibrate_855821_8.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:37 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:38 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:38 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:39 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:39 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:40 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 536 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fee4613ad48430a1985ae376578fcc78 with result from task 536
parsl.dataflow.dflow DEBUG: Task 536 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/07907354-9395-43ad-aaf6-7e3d152d0496_calibrate_855822_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/07907354-9395-43ad-aaf6-7e3d152d0496_calibrate_855822_27.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:40 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:41 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:42 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 386 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2f89d939019403603e8dca559d115f4d with result from task 386
parsl.dataflow.dflow DEBUG: Task 386 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/eea5d888-917e-4976-b9fa-1d40f242987d_calibrate_855821_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/eea5d888-917e-4976-b9fa-1d40f242987d_calibrate_855821_51.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:42 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 655 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 49b70cc98ca7adae1e4ae507022a2c14 with result from task 655
parsl.dataflow.dflow DEBUG: Task 655 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 656 has memoization hash d0ef94dba8c6158a96ec0fe8dd3a2fa9
parsl.dataflow.memoization INFO: Task 656 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 656 try 0 launched on executor multi with executor id 1116
parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/8aa86507-8c90-432b-99d4-bbd89a86e614_calibrate_855822_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/8aa86507-8c90-432b-99d4-bbd89a86e614_calibrate_855822_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/28d7671b-a642-4aa7-a1af-0f61db4d1708_characterizeImage_855822_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/28d7671b-a642-4aa7-a1af-0f61db4d1708_characterizeImage_855822_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:43 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:43 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4740821518abb94e7140f338e0317f61 with result from task 242
parsl.dataflow.dflow DEBUG: Task 242 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/235f360e-8b80-4577-9c31-f1fa17318f13_calibrate_855822_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/235f360e-8b80-4577-9c31-f1fa17318f13_calibrate_855822_20.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:44 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:45 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 1085 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 749111005d1d855cd336811ab6bd469f with result from task 1085
parsl.dataflow.dflow DEBUG: Task 1085 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/09eae4ed-c75d-48a0-9337-235b7b91ea74_calibrate_855822_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/09eae4ed-c75d-48a0-9337-235b7b91ea74_calibrate_855822_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:45 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 581 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c6a1bc5a569c31562725fb2b51d9ba64 with result from task 581
parsl.dataflow.dflow DEBUG: Task 581 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/a6c3b4bf-44c0-4b62-80c5-29d4aad869c8_calibrate_855822_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/a6c3b4bf-44c0-4b62-80c5-29d4aad869c8_calibrate_855822_51.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:46 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:47 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:47 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:48 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:48 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:49 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a18113f21462e7367a507d2747a842b6 with result from task 362
parsl.dataflow.dflow DEBUG: Task 362 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/5d9e038c-a18a-472e-9c97-e9b389f3ee05_calibrate_855821_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/5d9e038c-a18a-472e-9c97-e9b389f3ee05_calibrate_855821_7.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:50 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:50 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 65bad83ade58760f07933600af3f4d3b with result from task 2
parsl.dataflow.dflow DEBUG: Task 2 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/54638b1b-365c-4063-969a-1a732712af66_calibrate_855822_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/54638b1b-365c-4063-969a-1a732712af66_calibrate_855822_22.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:51 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:51 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8c8dec3e213f35f20591e17c387264bc with result from task 179
parsl.dataflow.dflow DEBUG: Task 179 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/cf7e54fa-1a7b-41b2-a4f6-6ece4bcb1148_calibrate_855821_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/cf7e54fa-1a7b-41b2-a4f6-6ece4bcb1148_calibrate_855821_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:52 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:53 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:53 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b3f81b15c72d5032dbbb89d0dcb5994a with result from task 334
parsl.dataflow.dflow DEBUG: Task 334 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 335 has memoization hash 215c4bae740f00559ce68867707c60d3
parsl.dataflow.memoization INFO: Task 335 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16aba520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 1117
parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/854901d7-2787-4a0d-8a25-bfd2665dccc8_calibrate_855822_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/854901d7-2787-4a0d-8a25-bfd2665dccc8_calibrate_855822_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/9e1bac5a-f371-45c8-8550-39c325de8911_characterizeImage_855822_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/9e1bac5a-f371-45c8-8550-39c325de8911_characterizeImage_855822_62.stderr
parsl.dataflow.dflow INFO: Task 737 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 54d147e3d1cec0780b38a9590f8b4f82 with result from task 737
parsl.dataflow.dflow DEBUG: Task 737 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f6916187-8ded-497d-b56b-3deaf46ed0bc_calibrate_855821_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f6916187-8ded-497d-b56b-3deaf46ed0bc_calibrate_855821_27.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:55 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:55 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:56 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 575 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 546322a06a3bc6274b62cbe1a30f5ab6 with result from task 575
parsl.dataflow.dflow DEBUG: Task 575 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/4600d054-8fe1-4a94-abea-c0112f97724d_calibrate_855822_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/4600d054-8fe1-4a94-abea-c0112f97724d_calibrate_855822_16.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:56 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:56 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:57 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:57 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 875 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7dfe323fd8f9c3819922c7da445c3e43 with result from task 875
parsl.dataflow.dflow DEBUG: Task 875 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/6b6c624c-871e-4408-80cd-fa9a38728faa_calibrate_855822_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/6b6c624c-871e-4408-80cd-fa9a38728faa_calibrate_855822_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:57 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:58 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:58 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:58 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:59 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:59 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 805 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b67a59e1b455bab70309210d572dbcfc with result from task 805
parsl.dataflow.dflow DEBUG: Task 805 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 806 has memoization hash 9ee0275e4138b6139c8b87bced421357
parsl.dataflow.memoization INFO: Task 806 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158951c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 806 try 0 launched on executor multi with executor id 1118
parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f4d86b75-bea5-4f8a-9dc6-a65baeda2d54_calibrate_855821_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f4d86b75-bea5-4f8a-9dc6-a65baeda2d54_calibrate_855821_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/178fe260-4ff9-496a-86e6-716a79e9429a_characterizeImage_855821_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/178fe260-4ff9-496a-86e6-716a79e9429a_characterizeImage_855821_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:59 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:01:59 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:00 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:00 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:00 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:01 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:01 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:01 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:02 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:02 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:02 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:02 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 79065de3b8dbd4bc011a5fa9bf99d622 with result from task 8
parsl.dataflow.dflow DEBUG: Task 8 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/549ae9fc-8e97-4f45-98bf-55796308ba35_calibrate_855822_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/549ae9fc-8e97-4f45-98bf-55796308ba35_calibrate_855822_13.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:03 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:03 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:03 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:04 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 422 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b63709df4ec3763f84e2275b44d3b939 with result from task 422
parsl.dataflow.dflow DEBUG: Task 422 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/2192cb69-a0e2-4e0f-8c1e-740b533904a7_calibrate_855821_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/2192cb69-a0e2-4e0f-8c1e-740b533904a7_calibrate_855821_35.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:04 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:04 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:05 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:05 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:05 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 434 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1a327002471d38f706e60f13b49a8929 with result from task 434
parsl.dataflow.dflow DEBUG: Task 434 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/214e4f66-8c1b-45d2-ad7c-5276f3c2eba5_calibrate_855821_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/214e4f66-8c1b-45d2-ad7c-5276f3c2eba5_calibrate_855821_20.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:05 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:06 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 557 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bceb78d5ecbee0b649c52fba08ff2eac with result from task 557
parsl.dataflow.dflow DEBUG: Task 557 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/8e6d41da-3052-492a-8b0d-179d8661465a_calibrate_855822_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/8e6d41da-3052-492a-8b0d-179d8661465a_calibrate_855822_35.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:06 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 607 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3f0e36882c759643db2edc6c4c899c66 with result from task 607
parsl.dataflow.dflow DEBUG: Task 607 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 608 has memoization hash 4570afee34d9d8fc951f66213e8825eb
parsl.dataflow.memoization INFO: Task 608 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 608 try 0 launched on executor multi with executor id 1119
parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/e6eb9862-b199-41a4-9d02-f0cc94b1d326_calibrate_855822_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/e6eb9862-b199-41a4-9d02-f0cc94b1d326_calibrate_855822_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/80ee0abb-6edb-479d-b69a-000e10a4d80d_characterizeImage_855822_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/80ee0abb-6edb-479d-b69a-000e10a4d80d_characterizeImage_855822_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:06 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:06 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:07 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:07 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:07 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:08 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:08 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:08 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:08 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:09 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:09 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d556a5474b2ce9d2ce85afc649ce26f1 with result from task 65
parsl.dataflow.dflow DEBUG: Task 65 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/157eaef5-ade8-40d2-8080-12dddb3d8a34_calibrate_855822_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/157eaef5-ade8-40d2-8080-12dddb3d8a34_calibrate_855822_11.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:09 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 75
parsl.dataflow.strategy DEBUG: Executor multi has 75 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:02:09 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:02:09 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 550 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 14e467cc7bd7a88f2bb9cf0aadeafe07 with result from task 550
parsl.dataflow.dflow DEBUG: Task 550 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 551 has memoization hash 019b37666f55cda8650affcbf474f005
parsl.dataflow.memoization INFO: Task 551 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1597c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 551 try 0 launched on executor multi with executor id 1120
parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/18444078-9c76-426f-a4ca-dea6c72024bd_calibrate_855821_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/18444078-9c76-426f-a4ca-dea6c72024bd_calibrate_855821_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/2b52761e-9b99-45d6-8c83-57d3861e9de5_characterizeImage_855821_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/2b52761e-9b99-45d6-8c83-57d3861e9de5_characterizeImage_855821_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:10 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:10 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:10 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:11 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 08bd387dcdd70d543a0aa01dc4508209 with result from task 125
parsl.dataflow.dflow DEBUG: Task 125 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/13648ddc-cad1-4090-879a-d0b11e750e50_calibrate_855822_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/13648ddc-cad1-4090-879a-d0b11e750e50_calibrate_855822_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:11 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:11 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:11 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:12 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:12 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:12 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:12 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:13 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:13 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 509 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1efb8fe8b7f3835dcfd97c2f493bcc5a with result from task 509
parsl.dataflow.dflow DEBUG: Task 509 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/b26af93f-3049-482c-afbd-200c54ceaefb_calibrate_855822_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/b26af93f-3049-482c-afbd-200c54ceaefb_calibrate_855822_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:13 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:13 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:14 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:14 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:14 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:14 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:15 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:15 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:15 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:16 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:16 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:16 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 673 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 41e5ee806d33b9dd2ae4a1aba279636d with result from task 673
parsl.dataflow.dflow DEBUG: Task 673 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 674 has memoization hash e553054d426e0bc68b8284206f1c36f8
parsl.dataflow.memoization INFO: Task 674 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa158087c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 674 try 0 launched on executor multi with executor id 1121
parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/a8ac31f1-ff8e-4c2a-971a-72ab3b2d7ef0_calibrate_855821_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/a8ac31f1-ff8e-4c2a-971a-72ab3b2d7ef0_calibrate_855821_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/f7f8474e-66ed-44b2-b7d2-0060ec832879_characterizeImage_855821_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/f7f8474e-66ed-44b2-b7d2-0060ec832879_characterizeImage_855821_49.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:16 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 012365f2e5af654349ff59f9e197da2f with result from task 149
parsl.dataflow.dflow DEBUG: Task 149 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/6304a29c-6516-4259-9ce7-5a8cddea13a5_calibrate_855822_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/6304a29c-6516-4259-9ce7-5a8cddea13a5_calibrate_855822_23.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:17 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:17 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:17 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 776 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4a6894c0dfe456194b3621cedf8b623a with result from task 776
parsl.dataflow.dflow DEBUG: Task 776 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/76828458-18a5-4282-8838-a8b21318cf05_calibrate_855822_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/76828458-18a5-4282-8838-a8b21318cf05_calibrate_855822_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:18 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:18 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:18 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.dataflow.dflow INFO: Task 596 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry be081424f9cc89ee5e5af34479b739a6 with result from task 596
parsl.dataflow.dflow DEBUG: Task 596 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f3f6ec6f-8880-42e0-94b9-6cd4f74d930f_calibrate_855821_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f3f6ec6f-8880-42e0-94b9-6cd4f74d930f_calibrate_855821_11.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:18 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:19 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:19 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 377 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c056081c46d37de121f4bc1fbbb2dc7a with result from task 377
parsl.dataflow.dflow DEBUG: Task 377 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/fcbc3290-4a2e-4b3b-8722-8427305c7e24_calibrate_855822_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/fcbc3290-4a2e-4b3b-8722-8427305c7e24_calibrate_855822_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:19 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:19 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:20 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:20 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:20 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:20 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:21 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:21 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1c55a737e223095eac4943e58a09a48e with result from task 281
parsl.dataflow.dflow DEBUG: Task 281 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/8185e7c2-a46e-4f9b-b043-0d8578a40bdb_calibrate_855821_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/8185e7c2-a46e-4f9b-b043-0d8578a40bdb_calibrate_855821_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:21 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:22 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:22 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:22 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:22 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:23 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:23 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:23 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:23 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:24 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:24 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 935 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 196c6d89c3943c7ee5a8eb227d1d9219 with result from task 935
parsl.dataflow.dflow DEBUG: Task 935 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/4a9e6ad6-c108-4237-8382-f4a7042d73ac_calibrate_855821_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/4a9e6ad6-c108-4237-8382-f4a7042d73ac_calibrate_855821_4.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:24 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:24 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:25 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:25 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 968 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b5cabe1b4def485838cd814637e94707 with result from task 968
parsl.dataflow.dflow DEBUG: Task 968 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/86e330e7-f7a8-423e-88ee-99ba88ed4dc0_calibrate_855822_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/86e330e7-f7a8-423e-88ee-99ba88ed4dc0_calibrate_855822_33.stderr
parsl.dataflow.dflow INFO: Task 869 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ed1c3363c41be493948de563e43538fe with result from task 869
parsl.dataflow.dflow DEBUG: Task 869 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/c8572556-02a7-457d-847b-80a5af32fc1f_calibrate_855822_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/c8572556-02a7-457d-847b-80a5af32fc1f_calibrate_855822_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:27 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:27 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:27 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:28 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:28 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:28 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:28 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:29 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:29 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:29 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1055 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4be219b04d5e7c680ef1cae86ba5f041 with result from task 1055
parsl.dataflow.dflow DEBUG: Task 1055 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/601e9b63-afa3-4748-8b0f-e1f65a4032ed_calibrate_855821_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/601e9b63-afa3-4748-8b0f-e1f65a4032ed_calibrate_855821_1.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:29 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:30 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:30 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 716 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9a19c98cb7f149cc74d8188ad18f381f with result from task 716
parsl.dataflow.dflow DEBUG: Task 716 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/a3080a4c-9d41-42fc-a5da-6a3f13036d5d_calibrate_855821_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/a3080a4c-9d41-42fc-a5da-6a3f13036d5d_calibrate_855821_14.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:30 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:30 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:31 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:31 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:31 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:31 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:32 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 06a50bd6d8d32657c9f4ae906eeddadf with result from task 194
parsl.dataflow.dflow DEBUG: Task 194 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/cf24b0f2-7ec7-4496-bbbe-98485f062929_calibrate_855822_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/cf24b0f2-7ec7-4496-bbbe-98485f062929_calibrate_855822_60.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:32 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:32 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:32 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:33 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:33 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:33 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:33 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:34 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:34 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:34 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:34 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c823ddd9abc13538e03d5d7df93aed6e with result from task 80
parsl.dataflow.dflow DEBUG: Task 80 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f82adf30-0d72-4925-a686-7040cdc694ee_calibrate_855821_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f82adf30-0d72-4925-a686-7040cdc694ee_calibrate_855821_16.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:35 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9c84d7ca4a951680114701fdadacc267 with result from task 223
parsl.dataflow.dflow DEBUG: Task 223 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 224 has memoization hash cda3d5662af2066234694a0ff738e623
parsl.dataflow.memoization INFO: Task 224 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a67100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 1122
parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/1e7c2e8e-03a4-4a30-8456-e9753ec44070_calibrate_855821_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/1e7c2e8e-03a4-4a30-8456-e9753ec44070_calibrate_855821_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/e17ff768-72f8-4967-91a7-905d931ecf88_characterizeImage_855821_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/e17ff768-72f8-4967-91a7-905d931ecf88_characterizeImage_855821_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:35 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 61
parsl.dataflow.strategy DEBUG: Executor multi has 61 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 61
parsl.dataflow.strategy DEBUG: Executor multi has 61 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.dflow INFO: Task 1019 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 254c4892a81b31a7f39ca51cb2e9f8a6 with result from task 1019
parsl.dataflow.dflow DEBUG: Task 1019 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/4175ceee-e704-4489-8005-7af750dff222_calibrate_855822_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/4175ceee-e704-4489-8005-7af750dff222_calibrate_855822_5.stderr
2024-03-09 21:02:40 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:02:40 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:41 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:41 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:41 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:41 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:42 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:42 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:42 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:42 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:43 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:43 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:43 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:43 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:44 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:44 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:44 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:44 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:44 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:45 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:45 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:45 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:45 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:46 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:46 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:46 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:46 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:47 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:47 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:47 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:47 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:48 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:48 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:48 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:48 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:49 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:49 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 785 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0d66338e7423208dbb627b6731b2fdd3 with result from task 785
parsl.dataflow.dflow DEBUG: Task 785 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/6f5464f8-3625-4389-bbc5-8ad76b83ff3a_calibrate_855821_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/6f5464f8-3625-4389-bbc5-8ad76b83ff3a_calibrate_855821_23.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:49 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:49 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:50 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:50 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:50 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:50 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 893 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3a5bddd8599c260610667af637439821 with result from task 893
parsl.dataflow.dflow DEBUG: Task 893 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/38362949-e20a-4cbd-970f-254f11acec28_calibrate_855821_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/38362949-e20a-4cbd-970f-254f11acec28_calibrate_855821_40.stderr
parsl.dataflow.dflow INFO: Task 380 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 684e170fd0c2b7c1a6b24b7593a6b83b with result from task 380
parsl.dataflow.dflow DEBUG: Task 380 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/576cf0de-e999-49a6-83a7-23d1ae8d5d3b_calibrate_855821_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/576cf0de-e999-49a6-83a7-23d1ae8d5d3b_calibrate_855821_48.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:51 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:51 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:51 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:51 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 1064 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e6a1575ef6fefaac00d25e74ffdcf389 with result from task 1064
parsl.dataflow.dflow DEBUG: Task 1064 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/ad0fb3b8-a99d-4de4-a400-de2a99dc4410_calibrate_855821_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/ad0fb3b8-a99d-4de4-a400-de2a99dc4410_calibrate_855821_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:52 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:52 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:52 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:52 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:53 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:53 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:53 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:53 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:53 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:54 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:54 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:54 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:54 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 920 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9cdbd77438a63bc09de5cff152ac516c with result from task 920
parsl.dataflow.dflow DEBUG: Task 920 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/f41daa24-9816-4b10-aa1e-fce85b17b6fc_calibrate_855822_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/f41daa24-9816-4b10-aa1e-fce85b17b6fc_calibrate_855822_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:55 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cfc2c80c64397f650268d8077cf2f4d2 with result from task 176
parsl.dataflow.dflow DEBUG: Task 176 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/3fb765e2-afc2-498d-b1e1-f0ed2f76c443_calibrate_855822_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/3fb765e2-afc2-498d-b1e1-f0ed2f76c443_calibrate_855822_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:55 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 689 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aa09892e5a389a694f75da78f33c35c4 with result from task 689
parsl.dataflow.dflow DEBUG: Task 689 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/fff2f9be-58ad-426a-b714-9b45a6d7bc2f_calibrate_855821_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/fff2f9be-58ad-426a-b714-9b45a6d7bc2f_calibrate_855821_41.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:55 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:55 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:56 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 926 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1c2d548ab328574600383b2e2a72f186 with result from task 926
parsl.dataflow.dflow DEBUG: Task 926 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/635a8d87-da35-4e85-883b-d6c08a880fe7_calibrate_855821_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/635a8d87-da35-4e85-883b-d6c08a880fe7_calibrate_855821_60.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:56 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:56 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 395 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1467dbc045f207c00e13f6c07fac6124 with result from task 395
parsl.dataflow.dflow DEBUG: Task 395 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/9e4eac21-5966-44bb-9207-a718fb4ccbcb_calibrate_855822_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/9e4eac21-5966-44bb-9207-a718fb4ccbcb_calibrate_855822_49.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:56 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 788 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 24b98d218d40c368816b98b2164234f4 with result from task 788
parsl.dataflow.dflow DEBUG: Task 788 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/3538f04c-4456-4716-af87-d03144e58fe4_calibrate_855821_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/3538f04c-4456-4716-af87-d03144e58fe4_calibrate_855821_29.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:57 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1010 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ba3fdcb62de7b3c6c06c47bd61ff9c22 with result from task 1010
parsl.dataflow.dflow DEBUG: Task 1010 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/b2fc135c-f1d6-405d-9db6-215b12f87138_calibrate_855822_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/b2fc135c-f1d6-405d-9db6-215b12f87138_calibrate_855822_29.stderr
parsl.dataflow.dflow INFO: Task 623 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a99d3eb691470226b12553fb1f834c32 with result from task 623
parsl.dataflow.dflow DEBUG: Task 623 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/74cd66d8-5bc4-45eb-8541-34da01549a0d_calibrate_855821_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/74cd66d8-5bc4-45eb-8541-34da01549a0d_calibrate_855821_57.stderr
parsl.dataflow.dflow INFO: Task 652 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 02f1ce6cca98a69d02af6a11c05ac0fd with result from task 652
parsl.dataflow.dflow DEBUG: Task 652 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 653 has memoization hash 32dc4789a6933463457bb714cbd939e9
parsl.dataflow.memoization INFO: Task 653 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa159c31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 653 try 0 launched on executor multi with executor id 1123
parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/d79e7c12-b3b6-4c39-9d8c-4b3a35bb606c_calibrate_855820_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/d79e7c12-b3b6-4c39-9d8c-4b3a35bb606c_calibrate_855820_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/bf49286d-cee7-44a0-8ec6-b91587dabdec_characterizeImage_855820_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855820/bf49286d-cee7-44a0-8ec6-b91587dabdec_characterizeImage_855820_61.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:59 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:59 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:59 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:02:59 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:00 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:00 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:00 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:00 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:01 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:01 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 75d3851465932ffe5f7bd5ea714f15a3 with result from task 287
parsl.dataflow.dflow DEBUG: Task 287 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/0d195381-7b2f-4aca-ad71-15cd43b22136_calibrate_855822_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/0d195381-7b2f-4aca-ad71-15cd43b22136_calibrate_855822_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:01 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 947 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e09680d998563915a0ea51ca128833f4 with result from task 947
parsl.dataflow.dflow DEBUG: Task 947 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/6641271d-7427-4544-9b47-2723096b4a7c_calibrate_855822_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/6641271d-7427-4544-9b47-2723096b4a7c_calibrate_855822_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:03 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1087 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 99763c6e495d1ce754bcc3ef8d3ee80b with result from task 1087
parsl.dataflow.dflow DEBUG: Task 1087 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1088 has memoization hash cd0e473d77d25bb6c5435d95163a15c7
parsl.dataflow.memoization INFO: Task 1088 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa157c7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1088 try 0 launched on executor multi with executor id 1124
parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/0620cb64-312b-43dc-a457-9e68a688a0e3_calibrate_855821_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/0620cb64-312b-43dc-a457-9e68a688a0e3_calibrate_855821_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/01412f1c-5ed0-49ad-87c6-74536c0cc35a_characterizeImage_855821_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/01412f1c-5ed0-49ad-87c6-74536c0cc35a_characterizeImage_855821_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:04 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:04 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:04 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:05 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:05 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:05 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:06 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:06 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:06 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:07 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:07 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:07 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 46
parsl.dataflow.strategy DEBUG: Executor multi has 46 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 46
parsl.dataflow.strategy DEBUG: Executor multi has 46 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 37af65d83481ab69e24e9f860ce1449f with result from task 86
parsl.dataflow.dflow DEBUG: Task 86 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/6c9ceaaa-b547-4c46-ac57-c8628c8b7c03_calibrate_855821_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/6c9ceaaa-b547-4c46-ac57-c8628c8b7c03_calibrate_855821_12.stderr
parsl.dataflow.dflow INFO: Task 392 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 88a5e12d16a8564b82a20c81155f28e4 with result from task 392
parsl.dataflow.dflow DEBUG: Task 392 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/9ceeb967-300a-42ca-9b39-8ca2418d9888_calibrate_855822_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/9ceeb967-300a-42ca-9b39-8ca2418d9888_calibrate_855822_43.stderr
2024-03-09 21:03:13 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:03:13 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:13 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:13 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 830 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 328de90be88149aa10552060e9132a0f with result from task 830
parsl.dataflow.dflow DEBUG: Task 830 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/934539a4-0a94-4872-b61d-13ce098d5217_calibrate_855821_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/934539a4-0a94-4872-b61d-13ce098d5217_calibrate_855821_47.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:14 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 458 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d03efc6c696cddd687c9f7039efa752e with result from task 458
parsl.dataflow.dflow DEBUG: Task 458 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/30cecf1c-51d3-4834-8c5a-0a79a6bc59b2_calibrate_855822_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/30cecf1c-51d3-4834-8c5a-0a79a6bc59b2_calibrate_855822_36.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:14 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:14 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:14 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:15 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:15 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:15 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:16 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:16 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:16 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:16 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 950 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b42d33be99e5d0f9a1ffb5ab8ed717d3 with result from task 950
parsl.dataflow.dflow DEBUG: Task 950 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/e330ef32-49e0-413a-b093-7de9377a85ad_calibrate_855821_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/e330ef32-49e0-413a-b093-7de9377a85ad_calibrate_855821_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:17 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:17 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 992 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dd66279ac0af0e750b13a396df39ef30 with result from task 992
parsl.dataflow.dflow DEBUG: Task 992 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/96610ef8-7691-4720-a081-102a0a92353c_calibrate_855822_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/96610ef8-7691-4720-a081-102a0a92353c_calibrate_855822_34.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:17 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 629 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8841841b24f159c361d0eb83eb03e745 with result from task 629
parsl.dataflow.dflow DEBUG: Task 629 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/d24b429a-01aa-4af9-a462-9e50436b5cb2_calibrate_855822_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/d24b429a-01aa-4af9-a462-9e50436b5cb2_calibrate_855822_7.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:18 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:18 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:18 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:19 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:19 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:19 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:19 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:20 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:20 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:20 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:20 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 1070 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1ce26dda1eab16199778920b16b51d04 with result from task 1070
parsl.dataflow.dflow DEBUG: Task 1070 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/a92bcac1-0e54-4470-882c-a0d22a2d54c7_calibrate_855821_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/a92bcac1-0e54-4470-882c-a0d22a2d54c7_calibrate_855821_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:21 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:21 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:21 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:21 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:22 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:22 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:22 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:22 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:23 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:23 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 755 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4ec8d1ee20092d238b933d7c842037f3 with result from task 755
parsl.dataflow.dflow DEBUG: Task 755 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/0a98e223-df1b-45e2-961e-e5ba45bd7f87_calibrate_855822_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/0a98e223-df1b-45e2-961e-e5ba45bd7f87_calibrate_855822_47.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:23 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:24 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:24 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:24 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:24 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:25 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:25 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:25 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:25 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:26 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:26 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:26 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:26 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:27 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:27 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 998 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8c74f894f290e288a4329cdaa91fd51f with result from task 998
parsl.dataflow.dflow DEBUG: Task 998 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/81bbb656-c2a0-4451-a1c5-6f8098500370_calibrate_855822_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/81bbb656-c2a0-4451-a1c5-6f8098500370_calibrate_855822_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:27 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:27 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:28 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:28 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:28 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:28 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 440 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6693a32c20e7d218a67fc07211bc6ac4 with result from task 440
parsl.dataflow.dflow DEBUG: Task 440 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/e75793fb-c2b4-445e-8426-271abcce49d5_calibrate_855821_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/e75793fb-c2b4-445e-8426-271abcce49d5_calibrate_855821_24.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:29 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1049 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f229545f88c40fc7e7400f0bbfc2f1d6 with result from task 1049
parsl.dataflow.dflow DEBUG: Task 1049 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/59cb3f1d-4616-45fd-be15-eda988d3a794_calibrate_855821_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/59cb3f1d-4616-45fd-be15-eda988d3a794_calibrate_855821_28.stderr
parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry da11ac0ef3a8a5643410f50f0051dc21 with result from task 59
parsl.dataflow.dflow DEBUG: Task 59 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/aef71b02-5c43-48b7-868a-6e1ea31c3d0e_calibrate_855822_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/aef71b02-5c43-48b7-868a-6e1ea31c3d0e_calibrate_855822_41.stderr
parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry abc2d2025d57bd18ce59b3d5f8ea1c08 with result from task 260
parsl.dataflow.dflow DEBUG: Task 260 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/cac2efd4-4499-442b-b283-2e0d5911687a_calibrate_855821_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/cac2efd4-4499-442b-b283-2e0d5911687a_calibrate_855821_38.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:31 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:31 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 812 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4470d2697b03849686cf425edd769fb3 with result from task 812
parsl.dataflow.dflow DEBUG: Task 812 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/28342909-b282-4234-8b8f-c2280049d84b_calibrate_855822_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/28342909-b282-4234-8b8f-c2280049d84b_calibrate_855822_12.stderr
parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6a6cb6d14441031b82ef13c625fba3b2 with result from task 299
parsl.dataflow.dflow DEBUG: Task 299 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/169c1aec-0fa4-489f-afc9-2f373222798c_calibrate_855821_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/169c1aec-0fa4-489f-afc9-2f373222798c_calibrate_855821_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:33 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:33 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 470 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 36f35a6a47e47baaa5a87c7a8b1f0511 with result from task 470
parsl.dataflow.dflow DEBUG: Task 470 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/1f180673-5a99-4a17-b050-c597327ba6cd_calibrate_855822_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/1f180673-5a99-4a17-b050-c597327ba6cd_calibrate_855822_30.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:34 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:34 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:34 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:34 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:35 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:35 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:35 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:35 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 566 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry da2d8d69acd7d836afd74c7c95e89cd7 with result from task 566
parsl.dataflow.dflow DEBUG: Task 566 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/3d2f7ae9-0347-45ba-9c72-5fd5afa5b102_calibrate_855822_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/3d2f7ae9-0347-45ba-9c72-5fd5afa5b102_calibrate_855822_53.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:36 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fa0c4b0e63f026ffea1225d6253fff30 with result from task 197
parsl.dataflow.dflow DEBUG: Task 197 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/b03016eb-66dd-4d6a-bae7-16cce60d9d3a_calibrate_855822_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/b03016eb-66dd-4d6a-bae7-16cce60d9d3a_calibrate_855822_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:36 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:36 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:36 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:37 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:37 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:37 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:37 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 842 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3e9a31038eab6155385cc5daffba15b6 with result from task 842
parsl.dataflow.dflow DEBUG: Task 842 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/0cb1aaa3-c51d-4f49-b76c-3a3ca8c8760c_calibrate_855821_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/0cb1aaa3-c51d-4f49-b76c-3a3ca8c8760c_calibrate_855821_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:38 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:38 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:38 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:38 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 30d59ab26d75bfd41ce09b034fa73091 with result from task 143
parsl.dataflow.dflow DEBUG: Task 143 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/229d98ce-1c3b-481b-a949-8e69e552806d_calibrate_855821_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/229d98ce-1c3b-481b-a949-8e69e552806d_calibrate_855821_30.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:39 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:39 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:39 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:39 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:40 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:40 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:40 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:40 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 25
parsl.dataflow.strategy DEBUG: Executor multi has 25 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 25
parsl.dataflow.strategy DEBUG: Executor multi has 25 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.dflow INFO: Task 863 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ecf69be460945284dbbc413fdc7f969c with result from task 863
parsl.dataflow.dflow DEBUG: Task 863 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/2ab6a07b-f047-4619-bc76-4d64d6d07af4_calibrate_855821_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/2ab6a07b-f047-4619-bc76-4d64d6d07af4_calibrate_855821_53.stderr
2024-03-09 21:03:46 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:03:46 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:46 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:46 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:46 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:47 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:47 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:47 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:47 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:48 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:48 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:48 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:48 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:49 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:49 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:49 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:49 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:50 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:50 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:50 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:50 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:50 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:51 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:51 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:51 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:51 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:52 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:52 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:52 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Task 806 completed (launched -> exec_done)
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.memoization DEBUG: Storing app cache entry 9ee0275e4138b6139c8b87bced421357 with result from task 806
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:52 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow DEBUG: Task 806 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f4d86b75-bea5-4f8a-9dc6-a65baeda2d54_calibrate_855821_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f4d86b75-bea5-4f8a-9dc6-a65baeda2d54_calibrate_855821_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:53 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:53 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:53 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e5f9162cef2077b471ab68226fc895b8 with result from task 53
parsl.dataflow.dflow DEBUG: Task 53 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/f4a1b476-e7b7-49a9-b165-47c0564a0733_calibrate_855822_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/f4a1b476-e7b7-49a9-b165-47c0564a0733_calibrate_855822_48.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:53 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:53 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:54 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:54 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:54 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:54 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:55 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:55 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:55 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:55 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:56 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:56 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:56 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:56 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:57 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:57 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:57 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:57 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:57 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:58 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:58 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:58 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:58 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:59 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:59 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:59 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:03:59 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 692 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fd08c71c7e8df1f193920e22ba645dbd with result from task 692
parsl.dataflow.dflow DEBUG: Task 692 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/6148ed34-f8b3-4751-b8a2-f17263f52d12_calibrate_855822_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/6148ed34-f8b3-4751-b8a2-f17263f52d12_calibrate_855822_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:00 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:00 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:00 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 938 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ac6ff164c371e2b243a2527e8b4deb36 with result from task 938
parsl.dataflow.dflow DEBUG: Task 938 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/ff1f6ee0-4186-421f-80a4-e60ecbc8b6da_calibrate_855822_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/ff1f6ee0-4186-421f-80a4-e60ecbc8b6da_calibrate_855822_24.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:00 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:01 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:01 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:01 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:02 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:02 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:02 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:02 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:03 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:03 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f6486daf4e50b1d6e94aa48316d9f520 with result from task 185
parsl.dataflow.dflow DEBUG: Task 185 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f3d7bb04-4231-4877-b038-1430451fbeca_calibrate_855821_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/f3d7bb04-4231-4877-b038-1430451fbeca_calibrate_855821_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:05 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 1061 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 330eb21d561957be0b65d9b1cfa006fb with result from task 1061
parsl.dataflow.dflow DEBUG: Task 1061 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/3e6c36ee-3974-4e47-9392-8a38cbed1d04_calibrate_855822_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/3e6c36ee-3974-4e47-9392-8a38cbed1d04_calibrate_855822_38.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:07 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:07 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 563 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry abe37a008c96542fdf956973e800fa21 with result from task 563
parsl.dataflow.dflow DEBUG: Task 563 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/ef8d9ca7-1c70-4ab6-89f7-25536e7f5879_calibrate_855821_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/ef8d9ca7-1c70-4ab6-89f7-25536e7f5879_calibrate_855821_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:08 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:09 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:09 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:09 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:10 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:10 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:10 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:10 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:11 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:11 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:11 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:11 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:12 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:12 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:12 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:12 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:13 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 17
parsl.dataflow.strategy DEBUG: Executor multi has 17 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 17
parsl.dataflow.strategy DEBUG: Executor multi has 17 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.dflow INFO: Task 653 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 32dc4789a6933463457bb714cbd939e9 with result from task 653
parsl.dataflow.dflow DEBUG: Task 653 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/d79e7c12-b3b6-4c39-9d8c-4b3a35bb606c_calibrate_855820_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855820/d79e7c12-b3b6-4c39-9d8c-4b3a35bb606c_calibrate_855820_61.stderr
2024-03-09 21:04:18 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
parsl.dataflow.dflow INFO: Task 605 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e2d0af72597aec457e0a9beff42f3177 with result from task 605
parsl.dataflow.dflow DEBUG: Task 605 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/aaa5c5d4-71ab-4638-a7ea-a78f488da735_calibrate_855822_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/aaa5c5d4-71ab-4638-a7ea-a78f488da735_calibrate_855822_54.stderr
2024-03-09 21:04:18 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 215c4bae740f00559ce68867707c60d3 with result from task 335
parsl.dataflow.dflow DEBUG: Task 335 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/854901d7-2787-4a0d-8a25-bfd2665dccc8_calibrate_855822_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/854901d7-2787-4a0d-8a25-bfd2665dccc8_calibrate_855822_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:18 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:18 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:19 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:19 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:19 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:19 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:20 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:20 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:20 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:21 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:21 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:21 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:21 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:22 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:22 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:22 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:22 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:23 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:23 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:23 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:23 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:24 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:24 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:24 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:24 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:25 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:25 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:25 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:25 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:26 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:26 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:26 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:26 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:27 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:27 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:27 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:27 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:28 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:28 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:28 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:28 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:28 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:29 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:29 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:29 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:29 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:30 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:30 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:30 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:30 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:31 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:31 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:31 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:31 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:32 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:32 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:32 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:32 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:33 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:33 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:33 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:33 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:33 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:34 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:34 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:34 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:34 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:34 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:35 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:35 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:35 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 770 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f98f1df3cedf7de1054aca98006e412b with result from task 770
parsl.dataflow.dflow DEBUG: Task 770 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/ee50eb3d-7f44-475b-9e12-fc0b468bb6df_calibrate_855821_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/ee50eb3d-7f44-475b-9e12-fc0b468bb6df_calibrate_855821_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:35 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:35 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:36 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:36 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:36 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 656 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d0ef94dba8c6158a96ec0fe8dd3a2fa9 with result from task 656
parsl.dataflow.dflow DEBUG: Task 656 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/8aa86507-8c90-432b-99d4-bbd89a86e614_calibrate_855822_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/8aa86507-8c90-432b-99d4-bbd89a86e614_calibrate_855822_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:36 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:36 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:37 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:37 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:37 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:37 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:38 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:38 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:38 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:38 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:38 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:40 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:41 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:41 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:41 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 12
parsl.dataflow.strategy DEBUG: Executor multi has 12 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 12
parsl.dataflow.strategy DEBUG: Executor multi has 12 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.dflow INFO: Task 428 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 58d17b27ec666faa36ef43c9d4ceb03a with result from task 428
parsl.dataflow.dflow DEBUG: Task 428 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/44f6712d-4ea8-4075-9450-4a22f2d0b55b_calibrate_855821_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/44f6712d-4ea8-4075-9450-4a22f2d0b55b_calibrate_855821_54.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 11
parsl.dataflow.strategy DEBUG: Executor multi has 11 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:04:51 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:04:51 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:52 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:52 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:52 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:52 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:52 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:53 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:53 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:53 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:53 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 551 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 019b37666f55cda8650affcbf474f005 with result from task 551
parsl.dataflow.dflow DEBUG: Task 551 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/18444078-9c76-426f-a4ca-dea6c72024bd_calibrate_855821_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/18444078-9c76-426f-a4ca-dea6c72024bd_calibrate_855821_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:53 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:54 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:54 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:54 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:54 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:55 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:55 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:55 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:55 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:55 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:56 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:56 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:56 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:56 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:57 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:57 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:57 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:57 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:57 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:58 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:58 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:58 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:58 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:58 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:59 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:59 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:59 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:59 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:04:59 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:00 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:00 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:00 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:00 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:00 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:01 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:01 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:01 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:01 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:01 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:02 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 608 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4570afee34d9d8fc951f66213e8825eb with result from task 608
parsl.dataflow.dflow DEBUG: Task 608 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/e6eb9862-b199-41a4-9d02-f0cc94b1d326_calibrate_855822_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/e6eb9862-b199-41a4-9d02-f0cc94b1d326_calibrate_855822_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:02 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:02 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:02 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:02 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:03 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:03 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:03 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:03 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:03 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:04 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:04 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:04 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:04 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:04 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:05 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:05 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:05 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:05 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:06 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:06 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:06 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:06 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:06 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:07 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:07 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:07 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:07 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:07 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:08 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:08 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:08 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:08 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:08 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:09 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:09 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:09 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:09 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:10 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:10 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:10 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:10 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:10 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 9
parsl.dataflow.strategy DEBUG: Executor multi has 9 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 9
parsl.dataflow.strategy DEBUG: Executor multi has 9 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 9
parsl.dataflow.strategy DEBUG: Executor multi has 9 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.dflow INFO: Task 674 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e553054d426e0bc68b8284206f1c36f8 with result from task 674
parsl.dataflow.dflow DEBUG: Task 674 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/a8ac31f1-ff8e-4c2a-971a-72ab3b2d7ef0_calibrate_855821_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/a8ac31f1-ff8e-4c2a-971a-72ab3b2d7ef0_calibrate_855821_49.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 8
parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:05:26 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:05:26 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:26 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:26 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:26 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:27 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:27 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:27 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:27 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:27 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:28 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:28 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:28 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:28 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:28 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:29 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:29 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:29 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:29 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:29 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:30 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:30 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:30 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:30 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:31 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:31 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:31 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:31 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:31 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:32 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:32 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:32 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:32 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:32 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:33 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:33 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:33 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:33 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cda3d5662af2066234694a0ff738e623 with result from task 224
parsl.dataflow.dflow DEBUG: Task 224 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/1e7c2e8e-03a4-4a30-8456-e9753ec44070_calibrate_855821_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/1e7c2e8e-03a4-4a30-8456-e9753ec44070_calibrate_855821_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:33 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:34 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:34 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:34 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:34 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:35 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:35 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:35 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 1088 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cd0e473d77d25bb6c5435d95163a15c7 with result from task 1088
parsl.dataflow.dflow DEBUG: Task 1088 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/0620cb64-312b-43dc-a457-9e68a688a0e3_calibrate_855821_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/0620cb64-312b-43dc-a457-9e68a688a0e3_calibrate_855821_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:35 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:35 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:36 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:36 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:36 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:36 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:36 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:37 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:37 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:37 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:37 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:37 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:38 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:38 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:38 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:38 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:39 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:39 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:39 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:39 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:39 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:40 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:40 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:40 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:40 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:40 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:41 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:41 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:41 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:41 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:41 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:42 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:42 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:42 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:42 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:42 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:43 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:45 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:45 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:45 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:46 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:46 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:46 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:46 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:46 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:47 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:47 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6
parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6
parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6
parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:05:57 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:05:57 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:57 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:57 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:58 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:58 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:58 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:58 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:58 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:59 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:59 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:59 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:05:59 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:00 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:00 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:00 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:00 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:00 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:01 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:01 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:01 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:01 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:01 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:02 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:02 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:02 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:02 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:02 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:03 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:03 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:03 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:03 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:03 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:04 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:04 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:04 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:04 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:04 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:05 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:05 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:05 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:05 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:05 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:06 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:06 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:06 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:06 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:06 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:07 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:07 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:07 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:07 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:07 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:08 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:08 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:08 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:08 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:09 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:09 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:09 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:09 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:09 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:10 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:10 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:10 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:10 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:10 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:11 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:11 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:11 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:11 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:12 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:12 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:12 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:12 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:12 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:13 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:13 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:13 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:13 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:13 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:14 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:14 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:14 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:14 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:15 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:15 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:15 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:15 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:15 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:18 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:18 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:18 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6
parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6
parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.dflow INFO: Task 904 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry da190e75fac59397d9d809b73dd5476e with result from task 904
parsl.dataflow.dflow DEBUG: Task 904 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 905 has memoization hash 0a6116b98ad1dc33427edd9f80e4cb42
parsl.dataflow.memoization INFO: Task 905 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa156eb920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 905 try 0 launched on executor multi with executor id 1125
parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/4838734a-3f72-4c87-8731-56081294d341_calibrate_855822_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/4838734a-3f72-4c87-8731-56081294d341_calibrate_855822_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/313bd6af-fee3-4ede-ab6b-527851c5b361_characterizeImage_855822_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855822/313bd6af-fee3-4ede-ab6b-527851c5b361_characterizeImage_855822_61.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6
parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:06:28 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:06:28 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:31 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:31 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:31 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:31 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:32 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:32 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:32 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:32 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:33 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:33 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:33 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:33 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:33 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:34 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:34 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:34 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:34 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:35 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:35 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:35 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:35 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:36 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ce667f0850ca86143dd441847af193fa with result from task 133
parsl.dataflow.dflow DEBUG: Task 133 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 134 has memoization hash fc1fc8ab457da087269f6e9b0259aa54
parsl.dataflow.memoization INFO: Task 134 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a41300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 1126
parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/fa18e155-0c02-42f8-93c1-203f2221a422_calibrate_855821_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/fa18e155-0c02-42f8-93c1-203f2221a422_calibrate_855821_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/0d47caae-2778-4361-af5c-f933cacb6268_characterizeImage_855821_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/0d47caae-2778-4361-af5c-f933cacb6268_characterizeImage_855821_34.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:36 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:36 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:36 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:36 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:37 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:37 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:37 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:37 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:38 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:38 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:38 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:38 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:39 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:39 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:39 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:39 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:39 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:40 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:40 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:40 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:40 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:41 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:41 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:41 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:41 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:41 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:42 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:42 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:42 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:42 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:43 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:43 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:43 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:43 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:43 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:44 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:44 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:44 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:44 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:44 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:45 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:45 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:45 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:45 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:45 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:46 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:46 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:46 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:46 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:46 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:47 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:47 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:47 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:47 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:47 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:48 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:48 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:48 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:48 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:48 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:49 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:49 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:49 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:49 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:50 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:50 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:50 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:50 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:06:50 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6
parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6
parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6
parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:07:01 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:07:01 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:02 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:02 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:02 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:03 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:03 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:04 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:04 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:04 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:04 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:04 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:05 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:05 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:05 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:05 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:06 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:06 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:06 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:06 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:06 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:07 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:07 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:07 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:07 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:07 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:08 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:08 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:08 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:08 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:08 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:09 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:09 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.dataflow.dflow INFO: Task 505 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2861f5d17b9c04c054eb9c03a8b9ad9b with result from task 505
parsl.dataflow.dflow DEBUG: Task 505 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 506 has memoization hash 927480ed0e904ba5d0942c04ef61b742
parsl.dataflow.memoization INFO: Task 506 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15945800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 506 try 0 launched on executor multi with executor id 1127
parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/60257eff-6c18-403f-86cb-c9e930a6eee6_calibrate_855821_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/60257eff-6c18-403f-86cb-c9e930a6eee6_calibrate_855821_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/1804422f-3cef-4925-bb30-b7ec8d29e1cd_characterizeImage_855821_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/1804422f-3cef-4925-bb30-b7ec8d29e1cd_characterizeImage_855821_61.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:09 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:09 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:09 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:10 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:10 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:10 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:10 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:10 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:11 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:11 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:11 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:11 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:11 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:12 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:12 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:12 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:12 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:12 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:13 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:13 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:13 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:13 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:13 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:14 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:14 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:14 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:14 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:14 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:15 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:15 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:15 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:15 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:15 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:16 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:16 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:16 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:16 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:17 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:17 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:17 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:17 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:17 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:18 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:18 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:18 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:18 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:18 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:19 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:19 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:19 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:19 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:19 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:20 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:20 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:20 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:20 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:20 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:21 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:21 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:21 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6
parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6
parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6
parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:07:31 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:07:31 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:34 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:34 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:34 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:34 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:34 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:35 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:35 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:35 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:35 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:35 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:36 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:36 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:36 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:36 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:36 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:37 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:37 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:37 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:37 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:37 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:38 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:38 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:38 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:38 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:38 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:39 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:39 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:39 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:39 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:40 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:40 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:40 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:40 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:40 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:41 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:41 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:41 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:41 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:41 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:42 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:42 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:42 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:42 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:42 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:43 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:43 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:43 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:43 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:43 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:44 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:44 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:44 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:44 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:44 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:45 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:45 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:45 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:45 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:45 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:46 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:46 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:46 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:46 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:46 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:47 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:47 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:47 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:47 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:47 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:48 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:48 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:48 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:48 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:48 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:49 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:49 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:49 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:49 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:49 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:50 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:50 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:50 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:50 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:50 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:51 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:51 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:51 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:51 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:51 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:52 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:07:52 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6
parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6
parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6
parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:08:02 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:08:02 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:02 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:03 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:03 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:05 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:05 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:05 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:06 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:06 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:06 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:07 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:07 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:07 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:07 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:07 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:08 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:08 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:08 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:08 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:08 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:09 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:09 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:09 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:09 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:09 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:10 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:10 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:10 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:10 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:10 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:11 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:11 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:11 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:11 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:11 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:12 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:12 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:12 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:12 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:13 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:13 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:13 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:13 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:13 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:14 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:14 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:14 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:14 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:14 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:15 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:15 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:15 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:15 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:15 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:16 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:16 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:16 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:16 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:16 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 905 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0a6116b98ad1dc33427edd9f80e4cb42 with result from task 905
parsl.dataflow.dflow DEBUG: Task 905 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/4838734a-3f72-4c87-8731-56081294d341_calibrate_855822_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855822/4838734a-3f72-4c87-8731-56081294d341_calibrate_855822_61.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:17 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:17 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:17 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:17 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:17 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:18 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:18 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:18 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:18 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:18 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:19 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:19 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:19 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:19 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:19 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:20 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:20 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:20 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:20 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:21 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:21 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:21 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:21 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:21 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:22 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:22 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:22 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:22 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:22 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:23 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:23 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:23 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:23 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5
parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5
parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5
parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.dflow INFO: Task 506 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 927480ed0e904ba5d0942c04ef61b742 with result from task 506
parsl.dataflow.dflow DEBUG: Task 506 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/60257eff-6c18-403f-86cb-c9e930a6eee6_calibrate_855821_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/60257eff-6c18-403f-86cb-c9e930a6eee6_calibrate_855821_61.stderr
2024-03-09 21:08:34 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:08:34 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:34 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:34 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:34 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:34 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:35 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:35 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:35 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:35 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:35 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:36 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:36 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:36 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:38 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:38 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:39 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:39 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:40 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:40 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:40 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:40 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:40 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:41 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:41 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:41 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:41 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:41 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:42 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:42 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:42 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fc1fc8ab457da087269f6e9b0259aa54 with result from task 134
parsl.dataflow.dflow DEBUG: Task 134 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/fa18e155-0c02-42f8-93c1-203f2221a422_calibrate_855821_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/fa18e155-0c02-42f8-93c1-203f2221a422_calibrate_855821_34.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:42 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:43 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:43 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:43 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:43 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:43 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:44 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:44 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:44 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:44 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:44 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:45 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:45 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:45 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:45 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:45 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:46 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:46 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:46 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:46 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:46 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:47 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:47 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:47 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:47 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:47 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:48 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:48 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:48 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:48 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:49 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:49 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:49 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:49 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:49 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:50 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:50 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:50 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:50 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:50 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:51 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:51 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:51 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:51 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:51 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:52 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:52 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:52 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:52 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:52 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:53 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:53 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:53 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:53 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:54 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:54 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:54 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:54 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:54 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:55 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:55 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:08:55 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:09:05 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:09:05 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:06 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:06 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:06 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:06 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:06 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:07 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:07 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:07 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:07 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:07 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:08 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:08 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:08 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:08 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:08 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:09 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:09 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:09 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:09 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:11 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:12 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:12 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:12 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:13 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:13 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:13 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:13 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:13 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:14 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:14 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:14 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:14 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:14 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:15 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:15 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:15 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:15 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:15 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:16 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:16 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:16 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:16 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:16 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:17 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:17 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:17 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:17 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:18 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:18 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:18 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:18 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:18 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:19 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:19 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:19 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:19 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:19 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:20 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:20 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:20 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:20 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:20 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:21 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:21 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:21 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:21 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:22 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:22 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:22 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:22 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:22 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:23 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:23 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:23 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:23 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:23 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:24 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:24 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:24 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:24 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:25 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:25 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:25 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:25 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:25 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:26 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:26 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:26 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:26 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:27 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:27 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:09:37 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:09:37 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:37 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:37 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:38 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:38 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:38 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:38 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:38 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:39 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:39 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:39 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:39 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:39 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:40 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:40 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:40 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:40 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:40 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:41 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:41 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:41 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:41 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:41 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:42 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:42 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:42 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:42 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:44 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:45 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:45 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:45 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:45 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:46 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:46 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:46 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:46 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:46 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:47 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:47 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:47 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:47 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:47 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:48 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:48 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:48 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:48 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:48 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:49 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:49 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:49 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:49 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:49 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:50 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:50 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:50 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:50 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:51 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:51 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:51 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:51 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:51 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:52 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:52 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:52 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:52 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:52 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:53 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:53 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:53 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:53 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:53 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:54 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:54 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:54 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:54 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:54 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:55 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:55 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:55 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:55 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:55 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:56 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:56 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:56 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:56 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:56 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:57 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:57 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:57 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:57 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:58 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:09:58 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:10:08 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:10:08 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:08 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:08 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:09 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:09 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:09 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:09 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:09 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:10 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:10 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:10 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:10 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:10 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:11 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:11 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:11 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:11 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:11 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:12 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:12 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:12 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:12 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:12 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:13 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:13 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:13 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:13 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:14 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:14 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:14 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:14 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:14 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:15 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:15 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:15 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:15 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:17 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:17 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:18 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:18 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:18 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:18 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:19 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:19 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:19 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:19 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:20 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:20 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:20 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:20 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:20 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:21 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:21 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:21 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:21 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:21 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:22 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:22 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:22 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:22 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:22 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:23 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:23 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:23 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:23 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:23 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:24 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:24 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:24 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:24 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:25 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:25 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:25 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:25 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:25 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:26 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:26 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:26 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:26 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:26 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:27 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:27 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:27 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:27 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:27 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:28 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:28 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:28 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:28 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:28 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:29 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:29 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:10:39 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:10:39 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:39 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:39 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:40 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:40 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:40 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:40 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:41 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:41 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:41 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:41 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:41 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:42 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:42 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:42 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:42 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:42 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:43 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:43 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:43 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:43 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:43 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:44 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:44 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:44 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:44 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:44 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:45 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:45 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:45 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:45 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:45 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:46 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:46 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:46 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:46 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:46 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:47 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:47 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:47 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:47 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:47 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:48 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:48 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:48 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:50 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:50 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:51 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:51 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:51 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:52 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:52 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:52 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:52 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:52 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:53 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:53 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:53 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:53 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:53 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:54 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:54 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:54 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:54 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:55 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:55 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:55 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:55 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Task 475 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ab15d2919918ea2d48c044d48c4d736c with result from task 475
parsl.dataflow.dflow DEBUG: Task 475 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 476 has memoization hash 0608e87714e842ef3da823a19a931a93
parsl.dataflow.memoization INFO: Task 476 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa1590f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:55 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.dataflow.dflow INFO: Parsl task 476 try 0 launched on executor multi with executor id 1128
parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/239c40b9-df91-4250-bbda-f7f0c042084a_calibrate_855821_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/239c40b9-df91-4250-bbda-f7f0c042084a_calibrate_855821_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/8d33728a-05d1-4f47-b28c-12afb1775c2e_characterizeImage_855821_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/8d33728a-05d1-4f47-b28c-12afb1775c2e_characterizeImage_855821_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:56 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:56 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:56 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:56 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:56 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:57 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:57 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:57 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:57 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:57 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:58 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:58 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:58 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:58 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:59 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:59 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:59 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:59 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:10:59 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:00 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:00 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:00 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:00 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0b75b0a1ebdb08d2b3cc0dc4bec87a8a with result from task 181
parsl.dataflow.dflow DEBUG: Task 181 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 182 has memoization hash f7c111f12484cfc13c72a0c017802ee9
parsl.dataflow.memoization INFO: Task 182 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa16a64540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 1129
parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/c1c5a702-07c3-4bc8-823d-1cb9e60af673_calibrate_855821_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/c1c5a702-07c3-4bc8-823d-1cb9e60af673_calibrate_855821_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/702b3551-ae2c-4941-a5f3-f54f3bb80bbf_characterizeImage_855821_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855821/702b3551-ae2c-4941-a5f3-f54f3bb80bbf_characterizeImage_855821_36.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:11:10 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:11:11 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:11 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:11 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:11 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:11 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:12 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:12 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:12 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:12 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:12 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:13 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:13 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:13 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:13 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:13 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:14 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:14 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:14 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:14 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:14 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:15 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:15 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:15 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:15 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:15 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:16 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:16 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:16 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:16 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:16 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:17 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:17 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:17 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:17 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:17 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:18 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:18 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:18 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:18 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:19 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:19 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:19 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:19 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:19 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:20 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:20 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:20 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:20 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:20 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:21 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:21 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:21 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:23 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:23 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:23 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:24 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:24 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:24 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:25 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:25 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:25 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:25 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:25 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:26 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:26 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:26 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:26 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:26 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:27 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:27 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:27 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:27 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:27 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:28 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:28 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:28 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:28 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:29 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:29 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:29 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:29 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:29 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:30 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:30 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:30 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:30 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:30 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:31 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:31 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:31 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:31 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:32 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:11:42 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:11:42 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:42 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:42 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:43 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:43 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:43 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:43 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:43 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:44 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:44 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:44 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:44 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:44 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:45 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:45 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:45 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:45 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:45 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:46 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:46 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:46 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:46 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:46 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:47 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:47 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:47 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:47 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:48 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:48 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:48 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:48 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:48 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:49 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:49 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:49 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:49 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:49 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:50 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:50 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:50 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:50 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:50 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:51 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:51 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:51 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:51 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:51 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:52 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:52 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:52 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:52 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:52 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:53 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:53 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:53 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:53 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:54 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:54 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:56 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:56 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:56 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:57 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:57 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:57 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:57 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:58 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:58 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:58 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:58 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:58 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:59 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:59 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:59 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:11:59 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:00 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:00 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:00 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:00 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:00 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:01 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:01 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:01 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:01 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:01 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:02 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:02 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:02 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:02 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:02 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:03 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:03 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:03 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:12:13 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:12:13 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:14 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:14 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:14 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:14 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:14 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:15 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:15 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:15 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:15 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:15 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:16 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:16 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:16 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:16 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:16 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:17 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:17 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:17 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:17 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:17 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:18 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:18 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:18 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:18 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:19 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:19 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:19 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:19 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:19 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:20 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:20 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:20 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:20 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:20 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:21 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:21 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:21 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:21 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:21 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:22 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:22 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:22 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:22 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:22 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:23 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:23 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:23 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:23 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:24 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:24 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:24 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:24 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:24 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:25 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:25 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:25 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:25 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:26 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:26 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:26 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:26 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:26 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:27 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:29 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:29 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:29 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:30 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:30 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:30 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:30 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:31 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:31 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:31 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:31 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:32 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:32 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:32 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:32 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:32 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:33 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:33 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:33 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:33 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:33 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:34 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:34 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:34 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:34 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:34 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:35 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:35 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.dflow INFO: Task 476 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0608e87714e842ef3da823a19a931a93 with result from task 476
parsl.dataflow.dflow DEBUG: Task 476 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/239c40b9-df91-4250-bbda-f7f0c042084a_calibrate_855821_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/239c40b9-df91-4250-bbda-f7f0c042084a_calibrate_855821_59.stderr
2024-03-09 21:12:45 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:12:45 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:45 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:46 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:46 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:46 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:46 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:46 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:47 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:47 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:47 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:47 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:47 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:48 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:48 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:48 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:48 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:48 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:49 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:49 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:49 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:49 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:49 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:50 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:50 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:50 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:50 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:51 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:51 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:51 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:51 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:51 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:52 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:52 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:52 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:52 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:52 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:53 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:53 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:53 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:53 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:53 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:54 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:54 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:54 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:54 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:54 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:55 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:55 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:55 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:55 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:55 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:56 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:56 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:56 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:56 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:56 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:57 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:57 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:57 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:57 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:57 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:58 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:58 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:58 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:58 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:59 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:59 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:59 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:59 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:12:59 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:00 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:02 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:02 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:02 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:03 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:03 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:03 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:03 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:03 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:04 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:04 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:04 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:04 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:04 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:05 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:05 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:05 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:05 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:06 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:06 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:06 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:06 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:13:16 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:13:16 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:17 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:17 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:17 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:17 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:17 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:18 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:18 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:18 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:18 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:19 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:19 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:19 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:19 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:19 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:20 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:20 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:20 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:20 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:20 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:21 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:21 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:21 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:21 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:21 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:22 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:22 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:22 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:22 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:22 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:23 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:23 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:23 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:23 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:23 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:24 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:24 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:24 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:24 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:25 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:25 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:25 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:25 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:25 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:26 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:26 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:26 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:26 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:26 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:27 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:27 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:27 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:27 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:27 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:28 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:28 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:28 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:28 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:28 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:29 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:29 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:29 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:29 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:29 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:30 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:30 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:30 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:30 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:31 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:31 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:31 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:31 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:31 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:32 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:32 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:32 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:32 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:33 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:35 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:35 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:35 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:35 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:36 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:36 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done)
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.memoization DEBUG: Storing app cache entry f7c111f12484cfc13c72a0c017802ee9 with result from task 182
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow DEBUG: Task 182 checkpointed
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/c1c5a702-07c3-4bc8-823d-1cb9e60af673_calibrate_855821_36.stdout
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855821/c1c5a702-07c3-4bc8-823d-1cb9e60af673_calibrate_855821_36.stderr
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:36 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:36 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:37 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:37 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:37 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:37 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:37 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:38 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:13:48 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:13:48 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:48 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:48 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:48 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:49 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:49 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:49 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:49 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:50 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:50 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:50 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:50 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:50 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:51 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:51 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:51 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:51 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:51 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:52 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:52 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:52 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:52 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:52 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:53 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:53 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:53 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:53 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:53 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:54 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:54 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:54 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:54 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:55 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:55 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:55 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:55 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:55 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:56 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:56 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:56 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:56 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:56 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:57 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:57 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:57 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:57 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:57 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:58 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:58 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:58 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:58 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:58 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:59 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:59 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:59 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:13:59 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:00 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:00 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:00 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:00 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:00 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:01 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:01 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:01 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:01 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:01 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:02 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:02 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:02 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:02 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:02 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:03 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:03 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:03 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:03 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:03 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:04 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:04 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:04 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:04 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:04 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:05 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:05 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:05 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:05 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:07 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:08 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:08 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:09 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:09 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:09 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:10 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:14:20 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:14:20 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:20 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:20 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:20 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:21 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:21 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:21 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:21 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:22 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:22 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:22 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:22 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:22 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:23 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:23 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:23 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:23 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:23 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:24 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:24 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:24 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:24 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:25 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:25 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:25 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:25 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:25 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:26 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:26 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:26 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:26 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:26 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:27 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:27 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:27 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:27 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:27 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:28 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:28 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:28 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:28 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:28 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:29 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:29 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:29 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:29 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:29 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:30 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:30 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:30 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:30 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:31 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:31 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:31 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:31 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:32 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:32 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:32 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:32 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:32 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:33 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:33 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:33 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:33 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:33 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:34 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:34 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:34 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:34 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:34 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:35 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:35 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:35 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:35 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:35 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:36 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:36 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:36 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:36 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:36 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:37 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:37 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:37 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:37 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:37 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:38 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:38 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:38 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:38 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:39 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:39 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:39 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:14:51 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:14:51 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:51 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:51 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:52 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:52 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:52 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:52 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:52 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:53 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:53 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:53 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:53 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:54 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:54 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:54 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:54 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:54 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:55 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:55 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:55 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:55 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:55 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:56 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:56 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:56 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:56 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:56 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:57 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:57 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:57 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:57 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:57 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:58 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:58 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:58 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:58 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:59 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:59 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:59 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:59 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:14:59 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:00 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:00 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:00 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:00 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:00 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:01 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:01 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:01 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:01 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:01 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:02 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:02 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:02 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:02 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:03 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:03 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:03 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:03 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:03 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:04 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:04 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:04 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:04 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:04 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:05 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:05 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:05 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:05 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:06 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:06 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:06 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:06 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:06 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:07 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:07 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:07 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:07 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:08 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:08 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:08 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:08 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:08 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:09 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:09 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:09 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:09 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:09 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:10 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:10 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:10 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:10 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:15:25 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:15:26 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:26 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:26 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:26 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:26 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:27 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:27 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:27 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:27 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:27 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:28 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:28 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:28 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:28 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:28 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:29 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:29 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:29 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:29 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:29 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:30 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:30 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:30 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:30 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:31 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:31 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:31 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:31 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:32 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:32 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:32 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:32 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:32 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:33 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:33 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:33 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:33 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:33 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:34 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:34 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:34 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:34 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:34 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:35 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:35 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:35 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:35 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:35 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:36 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:36 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:36 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:36 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:36 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:37 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:37 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:37 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:37 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:38 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:38 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:38 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:38 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:38 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:39 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:39 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:39 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:39 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:39 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:40 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:40 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:40 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:40 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:40 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:41 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:41 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:41 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:41 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:41 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:42 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:42 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:42 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:42 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:42 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:43 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:43 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:43 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:43 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:44 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:44 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:44 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:44 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:44 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:15:45 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:16:00 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:16:00 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:00 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:00 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:00 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:01 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:01 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:01 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:01 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:01 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:02 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:02 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:02 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:02 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:03 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:03 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:03 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:03 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:03 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:04 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:04 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:04 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:04 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:04 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:05 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:05 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:05 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:05 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:05 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:06 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:06 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:06 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:06 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:06 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:07 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:07 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:07 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:07 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:07 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:08 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:08 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:08 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:08 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:08 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:09 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:09 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:09 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:09 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:09 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:10 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:10 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:10 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:10 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:10 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:11 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:11 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:11 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:11 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:11 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:12 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:12 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:12 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:12 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:12 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:13 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:13 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:13 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:13 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:13 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:14 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:14 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:14 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:14 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:15 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:15 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:15 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:15 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:15 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:16 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:16 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:16 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:16 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:16 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:17 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:17 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:17 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:17 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:17 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:18 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:18 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:18 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:18 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:18 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:16:34 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:16:34 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:34 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:34 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:34 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:34 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:35 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:35 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:35 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:35 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:36 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:36 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:36 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:36 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:36 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:37 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:37 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:37 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:37 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:37 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:38 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:38 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:38 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:38 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:38 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:39 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:39 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:39 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:39 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:40 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:40 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:40 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:40 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:40 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:41 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:41 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:41 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:41 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:41 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:42 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:42 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:42 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:42 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:42 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:43 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:43 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:43 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:43 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:43 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:44 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:44 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:44 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:44 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:44 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:45 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:45 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:45 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:45 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:45 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:46 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:46 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:46 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:46 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:46 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:47 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:47 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:47 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:47 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:47 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:48 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:48 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:48 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:48 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:49 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:49 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:49 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:49 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:49 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:50 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:50 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:50 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:50 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:50 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:51 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:51 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:51 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:51 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:51 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:52 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:52 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:52 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:52 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:16:53 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:17:08 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:17:08 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:08 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:08 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:08 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:09 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:09 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:09 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:09 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:09 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:10 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:10 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:10 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:10 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:10 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:11 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:11 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:11 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:11 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:11 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:12 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:12 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:12 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:12 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:13 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:13 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:13 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:13 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:13 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:14 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:14 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:14 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:14 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:14 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:15 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:15 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:15 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:15 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:15 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:16 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:16 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:16 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:16 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:16 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:17 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:17 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:17 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:17 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:17 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:18 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:18 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:18 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:18 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:19 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:19 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:19 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:19 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:19 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:20 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:20 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:20 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:20 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:20 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:21 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:21 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:21 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:21 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:21 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:22 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:22 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:22 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:22 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:22 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:23 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:23 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:23 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:23 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:23 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:24 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:24 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:24 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:24 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:24 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:25 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:25 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:25 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:25 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:27 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:28 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:28 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:29 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:29 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:29 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:17:39 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:17:39 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:40 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:40 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:40 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:40 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:40 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:41 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:41 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:41 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:41 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:41 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:42 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:42 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:42 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:42 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:42 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:43 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:43 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:43 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:43 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:43 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:44 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:44 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:44 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:44 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:44 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:45 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:45 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:45 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:45 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:46 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:46 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:46 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:46 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:46 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:47 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:47 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:47 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:47 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:47 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:48 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:48 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:48 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:48 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:48 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:49 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:49 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:49 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:49 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:49 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:50 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:50 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:50 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:50 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:50 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:51 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:51 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:51 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:51 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:51 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:52 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:52 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:52 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:52 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:52 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:53 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:53 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:53 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:53 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:54 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:54 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:54 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:54 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:54 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:55 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:55 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:55 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:55 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:55 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:56 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:56 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:56 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:56 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:56 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:57 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:57 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:57 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:57 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:57 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:58 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:58 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:17:58 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:18:13 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:18:13 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:14 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:14 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:14 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:14 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:14 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:15 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:15 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:15 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:15 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:15 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:16 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:16 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:16 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:16 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:16 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:17 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:17 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:17 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:17 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:17 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:18 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:18 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:18 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:18 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:19 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:19 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:19 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:19 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:19 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:20 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:20 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:20 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:20 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:20 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:21 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:21 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:21 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:21 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:21 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:22 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:22 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:22 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:22 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:22 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:23 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:23 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:23 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:23 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:24 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:24 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:24 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:24 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:24 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:25 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:25 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:25 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:25 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:25 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:26 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:26 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:26 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:26 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:27 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:27 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:27 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:27 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:27 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:28 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:28 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:28 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:28 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:28 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:29 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:29 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:29 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:29 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:30 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:30 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:30 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:30 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:30 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:31 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:31 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:31 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:31 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:32 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:32 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:32 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:34 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:34 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:34 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:18:46 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:18:46 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:46 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:46 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:46 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:47 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:47 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:47 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:47 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:47 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:48 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:48 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:48 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:48 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:48 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:49 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:49 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:49 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:49 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:49 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:50 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:50 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:50 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:50 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:50 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:51 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:51 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:51 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:51 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:51 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:52 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:52 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:52 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:52 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:52 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:53 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:53 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:53 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:53 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:53 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:54 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:54 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:54 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:54 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:55 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:55 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:55 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:55 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:55 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:56 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:56 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:56 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:56 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:56 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:57 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:57 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:57 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:57 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:57 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:58 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:58 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:58 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:58 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:59 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:59 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:59 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:59 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:18:59 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:00 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:00 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:00 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:00 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:00 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:01 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:01 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:01 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:01 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:01 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:02 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:02 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:02 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:02 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:02 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:03 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:03 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:03 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:03 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:03 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:04 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:04 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:04 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:04 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:04 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:19:20 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:19:20 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:20 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:20 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:20 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:21 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:21 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:21 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:21 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:21 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:22 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:22 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:22 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:22 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:22 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:23 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:23 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:23 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:23 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:23 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:24 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:24 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:24 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:24 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:24 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:25 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:25 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:25 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:25 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:25 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:26 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:26 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:26 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:26 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:26 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:27 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:27 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:27 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:27 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:27 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:28 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:28 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:28 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:28 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:28 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:29 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:29 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:29 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:29 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:29 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:30 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:30 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:30 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:30 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:31 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:31 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:31 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:31 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:31 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:32 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:32 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:32 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:32 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:33 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:33 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:33 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:33 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:33 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:34 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:34 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:34 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:34 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:34 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:35 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:35 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:35 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:35 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:35 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:36 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:36 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:36 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:36 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:36 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:37 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:37 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:37 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:37 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:37 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:38 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:38 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:38 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:38 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:38 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:19:54 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:19:54 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:54 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:54 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:54 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:55 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:55 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:55 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:55 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:56 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:56 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:56 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:56 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:56 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:57 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:57 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:57 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:57 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:57 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:58 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:58 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:58 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:58 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:58 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:59 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:59 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:59 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:59 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:19:59 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:00 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:00 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:00 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:00 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:01 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:01 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:01 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:01 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:01 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:02 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:02 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:02 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:02 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:02 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:03 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:03 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:03 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:03 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:03 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:04 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:04 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:04 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:04 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:04 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:05 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:05 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:05 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:05 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:05 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:06 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:06 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:06 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:06 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:07 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:07 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:07 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:07 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:07 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:08 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:08 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:08 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:08 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:08 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:09 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:09 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:09 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:09 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:10 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:10 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:10 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:10 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:10 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:11 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:11 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:11 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:11 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:11 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:12 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:12 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:12 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:12 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:12 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:15 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:15 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:20:25 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:20:25 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:25 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:26 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:26 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:26 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:26 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:26 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:27 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:27 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:27 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:27 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:27 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:28 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:28 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:28 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:28 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:28 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:29 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:29 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:29 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:29 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:30 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:30 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:30 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:30 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:30 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:31 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:31 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:31 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:31 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:32 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:32 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:32 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:32 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:32 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:33 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:33 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:33 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:33 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:33 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:34 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:34 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:34 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:34 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:34 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:35 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:35 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:35 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:35 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:35 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:36 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:36 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:36 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:36 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:36 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:37 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:37 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:37 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:37 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:38 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:38 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:38 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:38 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:38 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:39 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:39 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:39 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:39 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:39 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:40 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:40 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:40 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:40 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:40 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:41 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:41 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:41 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:41 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:41 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:42 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:42 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:42 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:42 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:42 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:43 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:43 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:43 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:43 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:44 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:44 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:44 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:20:44 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:20:59 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:21:00 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:00 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:00 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:00 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:00 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:00 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:01 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:01 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:01 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:01 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:02 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:02 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:02 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:02 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:02 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.dataflow.dflow INFO: Task 739 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8f73dfb490880c4e831d6027bddb77e9 with result from task 739
parsl.dataflow.dflow DEBUG: Task 739 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 740 has memoization hash 513cc971237d7620660593a276aa9567
parsl.dataflow.memoization INFO: Task 740 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14aa15858cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 740 try 0 launched on executor multi with executor id 1130
parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/7d6a1690-a493-49e6-ab16-660cc0a7a244_calibrate_855909_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/7d6a1690-a493-49e6-ab16-660cc0a7a244_calibrate_855909_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/13685c98-bb4d-4402-96ca-6ed4012d88c4_characterizeImage_855909_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/characterizeImage/855909/13685c98-bb4d-4402-96ca-6ed4012d88c4_characterizeImage_855909_61.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:03 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:03 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:03 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:03 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:03 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:04 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:04 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:04 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:04 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:04 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:05 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:05 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:05 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:05 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:05 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:06 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:06 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:06 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:06 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:06 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:07 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:07 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:07 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:07 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:07 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:08 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:08 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:08 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:08 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:08 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:09 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:09 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:09 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:09 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:09 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:10 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:10 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:10 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:10 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:10 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:11 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:11 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:11 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:11 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:11 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:12 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:12 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:12 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:12 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:12 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:13 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:13 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:13 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:13 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:14 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:14 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:14 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:14 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:14 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:15 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:15 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:15 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:15 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:15 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:16 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:16 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:16 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:16 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:16 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:17 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:17 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:17 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:17 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:19 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:19 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:20 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:20 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:21:30 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:21:30 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:30 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:31 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:31 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:31 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:31 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:32 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:32 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:32 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:32 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:32 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:33 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:33 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:33 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:33 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:33 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:34 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:34 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:34 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:34 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:34 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:35 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:35 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:35 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:35 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:35 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:36 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:36 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:36 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:36 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:37 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:37 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:37 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:37 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:37 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:38 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:38 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:38 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:38 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:38 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:39 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:39 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:39 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:39 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:39 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:40 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:40 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:40 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:40 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:40 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:41 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:41 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:41 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:41 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:41 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:42 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:42 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:42 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:42 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:42 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:43 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:43 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:43 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:43 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:43 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:44 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:44 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:44 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:44 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:44 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:45 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:45 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:45 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:45 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:46 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:46 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:46 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:46 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:46 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:47 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:47 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:47 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:47 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:47 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:48 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:48 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:48 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:48 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:48 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:49 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:49 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:21:49 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
2024-03-09 21:22:04 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:22:04 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:05 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:05 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:05 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:05 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:05 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:06 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:06 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:06 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:06 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:06 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:07 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:07 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:07 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:07 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:07 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:08 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:08 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:08 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:08 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:08 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:09 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:09 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:09 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:09 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:09 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:10 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:10 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:10 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:10 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968884,16968885,16968886,16968887,16968888,16968889,16968890,16968891,16968892,16968893,16968894,16968895,16968896,16968897,16968898,16968899,16968900,16968901,16968902,16968903,16968904,16968905,16968906,16968907,16968909,16968911,16968912,16968913,16968917,16968918'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968917 R
16968918 R
16968907 R
16968909 R
16968911 R
16968912 R
16968913 R
16968899 R
16968900 R
16968901 R
16968902 R
16968903 R
16968904 R
16968905 R
16968906 R
16968893 R
16968894 R
16968895 R
16968896 R
16968897 R
16968898 R
16968887 R
16968888 R
16968889 R
16968890 R
16968891 R
16968892 R
16968884 R
16968885 R
16968886 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968907 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968911 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968886 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:10 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:11 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:11 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:11 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:11 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968919,16968920,16968921,16968922,16968923'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968919 R
16968920 R
16968921 R
16968922 R
16968923 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968921 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968923 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:12 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:12 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:12 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:12 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.dataflow.dflow INFO: Task 740 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 513cc971237d7620660593a276aa9567 with result from task 740
parsl.dataflow.dflow INFO: DFK cleanup initiated
parsl.dataflow.dflow INFO: Summary of tasks in DFK:
parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0
parsl.dataflow.dflow INFO: Tasks in state States.pending: 0
parsl.dataflow.dflow INFO: Tasks in state States.running: 0
parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 976
parsl.dataflow.dflow INFO: Tasks in state States.failed: 77
parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 63
parsl.dataflow.dflow INFO: Tasks in state States.launched: 0
parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0
parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0
parsl.dataflow.dflow INFO: Tasks in state States.joining: 0
parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0
parsl.dataflow.dflow INFO: End of summary
parsl.dataflow.dflow DEBUG: Task 740 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/7d6a1690-a493-49e6-ab16-660cc0a7a244_calibrate_855909_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/submit/DEEP/20190507/A1c/science#step1/20240309T205315Z/logs/calibrate/855909/7d6a1690-a493-49e6-ab16-660cc0a7a244_calibrate_855909_61.stderr
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Closing job status poller
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:12 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:13 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:13 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:13 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:13 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:13 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:14 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:14 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:14 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:14 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:14 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:15 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:15 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:15 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:15 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:15 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968924,16968925,16968926,16968927,16968928,16968929,16968930,16968931,16968932,16968933,16968934,16968935,16968936,16968937,16968938,16968939,16968940,16968941,16968942,16968943'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968936 R
16968937 R
16968938 R
16968939 R
16968940 R
16968941 R
16968942 R
16968943 R
16968930 R
16968931 R
16968932 R
16968933 R
16968934 R
16968935 R
16968925 R
16968926 R
16968927 R
16968928 R
16968929 R
16968924 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968936 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968937 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968938 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968939 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968940 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968941 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968942 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968943 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968930 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968931 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968932 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968933 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968934 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968935 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968925 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968926 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968927 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968928 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968929 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968924 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:16 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:16 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:16 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:16 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:16 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:17 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:17 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:17 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:17 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:18 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:18 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:18 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:18 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:18 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:19 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:19 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:19 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:19 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:19 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:20 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:20 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:20 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:20 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:20 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:21 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:21 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:21 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:21 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:22 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:22 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:22 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:24 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:24 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:25 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:25 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:25 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
2024-03-09 21:22:25 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968944,16968945,16968946,16968947,16968948,16968949,16968950,16968951,16968952,16968953,16968954,16968955,16968956,16968957,16968958,16968959,16968960,16968961,16968962,16968963,16968964,16968965,16968966,16968967,16968968,16968969,16968970,16968971,16968972,16968973,16968974,16968975,16968976,16968977,16968978,16968979,16968980'
parsl.providers.slurm.slurm DEBUG: squeue returned 16968944 R
16968945 R
16968946 R
16968947 R
16968948 R
16968949 R
16968950 R
16968951 R
16968952 R
16968953 R
16968954 R
16968955 R
16968956 R
16968957 R
16968958 R
16968959 R
16968960 R
16968961 R
16968962 R
16968963 R
16968964 R
16968965 R
16968966 R
16968967 R
16968968 R
16968969 R
16968970 R
16968971 R
16968972 R
16968973 R
16968974 R
16968975 R
16968976 R
16968977 R
16968978 R
16968979 R
16968980 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16968944 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968945 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968946 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968947 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968948 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968949 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968950 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968951 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968952 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968953 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968954 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968955 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968956 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968957 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968958 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968959 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968960 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968961 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968962 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968963 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968964 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968965 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968966 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968967 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968968 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968969 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968970 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968971 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968972 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968973 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968974 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968975 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968976 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968977 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968978 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968979 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16968980 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 0
parsl.dataflow.strategy DEBUG: Executor multi has 0 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 1: Executor has no active tasks
parsl.dataflow.strategy DEBUG: Strategy case 1b: Executor has no active tasks, and more (93) than minimum blocks (0)
parsl.dataflow.strategy DEBUG: Starting idle timer for executor. If idle time exceeds 120.0s, blocks will be scaled in
parsl.dataflow.strategy DEBUG: Idle time 3.814697265625e-06s is less than max_idletime 120.0s for executor multi; not scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22720767152208
parsl.dataflow.dflow INFO: Terminated job status poller
parsl.dataflow.dflow INFO: Scaling in and shutting down executors
parsl.dataflow.dflow INFO: Scaling in executor multi
parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=93, block_ids=[]
parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 93 blocks
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bbe5e3583140
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bbe5e3583140
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b93e8084a623
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b93e8084a623
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ce2efbd8ab1e
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ce2efbd8ab1e
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0e6d0ab7d9cf
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0e6d0ab7d9cf
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 077878c89870
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 077878c89870
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 27ce72710d36
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 27ce72710d36
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 27bf47b21d8b
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 27bf47b21d8b
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3a3c7ca8ddbb
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3a3c7ca8ddbb
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 84a06318ffed
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 84a06318ffed
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 204d1f39f896
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 204d1f39f896
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 57f6f7a59016
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 57f6f7a59016
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d2a0554f2d78
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d2a0554f2d78
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3afc83bb65a8
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3afc83bb65a8
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0004ce27c4d4
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0004ce27c4d4
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 27686495cc1c
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 27686495cc1c
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5d2dd7c6ebfe
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5d2dd7c6ebfe
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 81f8b097af3d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 81f8b097af3d
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e73024875f02
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e73024875f02
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4d9d2b743ae7
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4d9d2b743ae7
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f7f089452a90
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f7f089452a90
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 47728cdecb21
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 47728cdecb21
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 41774aa7117d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 41774aa7117d
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a878a3ad4efa
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a878a3ad4efa
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: acbedc52d02e
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: acbedc52d02e
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 15e2ae5a9def
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 15e2ae5a9def
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f9b806bcf5d8
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f9b806bcf5d8
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: deeb33a8429a
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: deeb33a8429a
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b1bc37e9ce38
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b1bc37e9ce38
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 08e03b2f7958
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 08e03b2f7958
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c20f7a8b0bb1
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c20f7a8b0bb1
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7fd7df2ef295
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7fd7df2ef295
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c97af23abfcd
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c97af23abfcd
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 61959f50804c
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 61959f50804c
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6d8ee5749aee
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6d8ee5749aee
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 73f7e650ebbd
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 73f7e650ebbd
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f3fcb17b3bc6
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f3fcb17b3bc6
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 38b6e9c6a985
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 38b6e9c6a985
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3a8c062faae5
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3a8c062faae5
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 58384e0c69b9
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 58384e0c69b9
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dfcf1036d149
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dfcf1036d149
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 552e0f790be7
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 552e0f790be7
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9c2a883f8c9c
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9c2a883f8c9c
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: eb94fa565ace
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: eb94fa565ace
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 447c2f092b4e
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 447c2f092b4e
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e8c1ea04e438
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e8c1ea04e438
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2389cbbb3413
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2389cbbb3413
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8180d5731786
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8180d5731786
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0c0b7ad87d89
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0c0b7ad87d89
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2856debdfeaf
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2856debdfeaf
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 87f3097e1e5a
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 87f3097e1e5a
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e5841624f318
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e5841624f318
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f429c1561731
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f429c1561731
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ffdf81fef198
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ffdf81fef198
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 57139875e3c7
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 57139875e3c7
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2c693161a43d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2c693161a43d
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9d687b0d38fe
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9d687b0d38fe
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 67ab637e7f1c
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 67ab637e7f1c
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fc3dcbadc230
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fc3dcbadc230
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 40e76d5c32ce
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 40e76d5c32ce
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 15c5b8c6d61c
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 15c5b8c6d61c
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7e534d03e763
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7e534d03e763
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f0f0ba0282ed
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f0f0ba0282ed
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3fab07bcbbe7
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3fab07bcbbe7
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5e423c0475b6
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5e423c0475b6
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2a5cdbb0bbd1
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2a5cdbb0bbd1
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 64ceec2e5d89
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 64ceec2e5d89
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2ca00ac81120
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2ca00ac81120
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2995432fe48a
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2995432fe48a
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4d9a597e1817
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4d9a597e1817
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1d91b33c4d1f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1d91b33c4d1f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3022137bd990
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3022137bd990
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a6d320e5fe2d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a6d320e5fe2d
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3602ea8a9670
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3602ea8a9670
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0370ab7c26b2
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0370ab7c26b2
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a5cf9bb9ab28
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a5cf9bb9ab28
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 332e4996597d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 332e4996597d
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: df5b5ed603a0
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: df5b5ed603a0
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4a98c87b67c2
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4a98c87b67c2
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 475c145f60d3
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 475c145f60d3
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 92f1fdb65fd4
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 92f1fdb65fd4
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8fdf2fd38e77
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8fdf2fd38e77
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b81ab6f311f7
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b81ab6f311f7
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d38a4ae1589b
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d38a4ae1589b
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b44285c5ad11
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b44285c5ad11
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cc130a680675
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cc130a680675
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a4e3e6d86920
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a4e3e6d86920
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c27cb1de9494
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c27cb1de9494
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a335992b64b3
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a335992b64b3
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7e32ee54b461
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7e32ee54b461
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bf5ef1e130bc
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bf5ef1e130bc
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bcaac31ba9ad
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bcaac31ba9ad
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 66f4272cff00
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 66f4272cff00
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 90fc040a394c
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 90fc040a394c
2024-03-09 21:22:26 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
2024-03-09 21:22:26 proc_lsst.multi:201 [INFO]  cancelling 16968931 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968931 on provider escience_a40
2024-03-09 21:22:26 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
2024-03-09 21:22:26 proc_lsst.multi:201 [INFO]  cancelling 16968979 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968979 on provider checkpoint
2024-03-09 21:22:26 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
2024-03-09 21:22:26 proc_lsst.multi:201 [INFO]  cancelling 16968909 on provider astro
proc_lsst.multi INFO: cancelling 16968909 on provider astro
2024-03-09 21:22:26 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
2024-03-09 21:22:26 proc_lsst.multi:201 [INFO]  cancelling 16968957 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968957 on provider checkpoint
2024-03-09 21:22:26 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
2024-03-09 21:22:26 proc_lsst.multi:201 [INFO]  cancelling 16968927 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968927 on provider escience_a40
2024-03-09 21:22:26 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
2024-03-09 21:22:26 proc_lsst.multi:201 [INFO]  cancelling 16968913 on provider astro
proc_lsst.multi INFO: cancelling 16968913 on provider astro
2024-03-09 21:22:27 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
2024-03-09 21:22:27 proc_lsst.multi:201 [INFO]  cancelling 16968933 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968933 on provider escience_a40
2024-03-09 21:22:27 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
2024-03-09 21:22:27 proc_lsst.multi:201 [INFO]  cancelling 16968959 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968959 on provider checkpoint
2024-03-09 21:22:27 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
2024-03-09 21:22:27 proc_lsst.multi:201 [INFO]  cancelling 16968945 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968945 on provider checkpoint
2024-03-09 21:22:27 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
2024-03-09 21:22:27 proc_lsst.multi:201 [INFO]  cancelling 16968969 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968969 on provider checkpoint
2024-03-09 21:22:27 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
2024-03-09 21:22:27 proc_lsst.multi:201 [INFO]  cancelling 16968952 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968952 on provider checkpoint
2024-03-09 21:22:27 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
2024-03-09 21:22:27 proc_lsst.multi:201 [INFO]  cancelling 16968974 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968974 on provider checkpoint
2024-03-09 21:22:28 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
2024-03-09 21:22:28 proc_lsst.multi:201 [INFO]  cancelling 16968934 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968934 on provider escience_a40
2024-03-09 21:22:28 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
2024-03-09 21:22:28 proc_lsst.multi:201 [INFO]  cancelling 16968888 on provider astro
proc_lsst.multi INFO: cancelling 16968888 on provider astro
2024-03-09 21:22:28 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
2024-03-09 21:22:28 proc_lsst.multi:201 [INFO]  cancelling 16968954 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968954 on provider checkpoint
2024-03-09 21:22:28 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
2024-03-09 21:22:28 proc_lsst.multi:201 [INFO]  cancelling 16968944 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968944 on provider checkpoint
2024-03-09 21:22:28 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
2024-03-09 21:22:28 proc_lsst.multi:201 [INFO]  cancelling 16968939 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968939 on provider escience_a40
2024-03-09 21:22:29 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
2024-03-09 21:22:29 proc_lsst.multi:201 [INFO]  cancelling 16968912 on provider astro
proc_lsst.multi INFO: cancelling 16968912 on provider astro
2024-03-09 21:22:29 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
2024-03-09 21:22:29 proc_lsst.multi:201 [INFO]  cancelling 16968968 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968968 on provider checkpoint
2024-03-09 21:22:29 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
2024-03-09 21:22:29 proc_lsst.multi:201 [INFO]  cancelling 16968946 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968946 on provider checkpoint
2024-03-09 21:22:29 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
2024-03-09 21:22:29 proc_lsst.multi:201 [INFO]  cancelling 16968921 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16968921 on provider escience_rtx6k
2024-03-09 21:22:29 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
2024-03-09 21:22:29 proc_lsst.multi:201 [INFO]  cancelling 16968887 on provider astro
proc_lsst.multi INFO: cancelling 16968887 on provider astro
2024-03-09 21:22:29 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
2024-03-09 21:22:29 proc_lsst.multi:201 [INFO]  cancelling 16968953 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968953 on provider checkpoint
2024-03-09 21:22:30 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
2024-03-09 21:22:30 proc_lsst.multi:201 [INFO]  cancelling 16968962 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968962 on provider checkpoint
2024-03-09 21:22:30 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
2024-03-09 21:22:30 proc_lsst.multi:201 [INFO]  cancelling 16968976 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968976 on provider checkpoint
2024-03-09 21:22:30 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
2024-03-09 21:22:30 proc_lsst.multi:201 [INFO]  cancelling 16968903 on provider astro
proc_lsst.multi INFO: cancelling 16968903 on provider astro
2024-03-09 21:22:30 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
2024-03-09 21:22:30 proc_lsst.multi:201 [INFO]  cancelling 16968900 on provider astro
proc_lsst.multi INFO: cancelling 16968900 on provider astro
2024-03-09 21:22:31 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
2024-03-09 21:22:31 proc_lsst.multi:201 [INFO]  cancelling 16968923 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16968923 on provider escience_rtx6k
2024-03-09 21:22:31 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
2024-03-09 21:22:31 proc_lsst.multi:201 [INFO]  cancelling 16968977 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968977 on provider checkpoint
2024-03-09 21:22:31 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
2024-03-09 21:22:31 proc_lsst.multi:201 [INFO]  cancelling 16968956 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968956 on provider checkpoint
2024-03-09 21:22:31 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
2024-03-09 21:22:31 proc_lsst.multi:201 [INFO]  cancelling 16968975 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968975 on provider checkpoint
2024-03-09 21:22:31 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
2024-03-09 21:22:31 proc_lsst.multi:201 [INFO]  cancelling 16968894 on provider astro
proc_lsst.multi INFO: cancelling 16968894 on provider astro
2024-03-09 21:22:32 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
2024-03-09 21:22:32 proc_lsst.multi:201 [INFO]  cancelling 16968901 on provider astro
proc_lsst.multi INFO: cancelling 16968901 on provider astro
2024-03-09 21:22:32 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
2024-03-09 21:22:32 proc_lsst.multi:201 [INFO]  cancelling 16968951 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968951 on provider checkpoint
2024-03-09 21:22:32 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
2024-03-09 21:22:32 proc_lsst.multi:201 [INFO]  cancelling 16968911 on provider astro
proc_lsst.multi INFO: cancelling 16968911 on provider astro
2024-03-09 21:22:32 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
2024-03-09 21:22:32 proc_lsst.multi:201 [INFO]  cancelling 16968886 on provider astro
proc_lsst.multi INFO: cancelling 16968886 on provider astro
2024-03-09 21:22:32 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
2024-03-09 21:22:32 proc_lsst.multi:201 [INFO]  cancelling 16968932 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968932 on provider escience_a40
2024-03-09 21:22:33 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
2024-03-09 21:22:33 proc_lsst.multi:201 [INFO]  cancelling 16968973 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968973 on provider checkpoint
2024-03-09 21:22:33 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
2024-03-09 21:22:33 proc_lsst.multi:201 [INFO]  cancelling 16968898 on provider astro
proc_lsst.multi INFO: cancelling 16968898 on provider astro
2024-03-09 21:22:33 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
2024-03-09 21:22:33 proc_lsst.multi:201 [INFO]  cancelling 16968967 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968967 on provider checkpoint
2024-03-09 21:22:33 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
2024-03-09 21:22:33 proc_lsst.multi:201 [INFO]  cancelling 16968955 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968955 on provider checkpoint
2024-03-09 21:22:33 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
2024-03-09 21:22:33 proc_lsst.multi:201 [INFO]  cancelling 16968950 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968950 on provider checkpoint
2024-03-09 21:22:34 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
2024-03-09 21:22:34 proc_lsst.multi:201 [INFO]  cancelling 16968980 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968980 on provider checkpoint
2024-03-09 21:22:34 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
2024-03-09 21:22:34 proc_lsst.multi:201 [INFO]  cancelling 16968918 on provider astro
proc_lsst.multi INFO: cancelling 16968918 on provider astro
2024-03-09 21:22:34 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
2024-03-09 21:22:34 proc_lsst.multi:201 [INFO]  cancelling 16968942 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968942 on provider escience_a40
2024-03-09 21:22:34 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
2024-03-09 21:22:34 proc_lsst.multi:201 [INFO]  cancelling 16968958 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968958 on provider checkpoint
2024-03-09 21:22:34 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
2024-03-09 21:22:34 proc_lsst.multi:201 [INFO]  cancelling 16968924 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968924 on provider escience_a40
2024-03-09 21:22:35 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
2024-03-09 21:22:35 proc_lsst.multi:201 [INFO]  cancelling 16968919 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16968919 on provider escience_rtx6k
2024-03-09 21:22:35 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
2024-03-09 21:22:35 proc_lsst.multi:201 [INFO]  cancelling 16968884 on provider astro
proc_lsst.multi INFO: cancelling 16968884 on provider astro
2024-03-09 21:22:35 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
2024-03-09 21:22:35 proc_lsst.multi:201 [INFO]  cancelling 16968970 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968970 on provider checkpoint
2024-03-09 21:22:35 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
2024-03-09 21:22:35 proc_lsst.multi:201 [INFO]  cancelling 16968949 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968949 on provider checkpoint
2024-03-09 21:22:35 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
2024-03-09 21:22:35 proc_lsst.multi:201 [INFO]  cancelling 16968906 on provider astro
proc_lsst.multi INFO: cancelling 16968906 on provider astro
2024-03-09 21:22:35 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
2024-03-09 21:22:35 proc_lsst.multi:201 [INFO]  cancelling 16968947 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968947 on provider checkpoint
2024-03-09 21:22:36 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:22:36 proc_lsst.multi:201 [INFO]  cancelling 51896 on provider local
proc_lsst.multi INFO: cancelling 51896 on provider local
parsl.providers.local.local DEBUG: Terminating job/proc_id: 51896
2024-03-09 21:22:36 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
2024-03-09 21:22:36 proc_lsst.multi:201 [INFO]  cancelling 16968963 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968963 on provider checkpoint
2024-03-09 21:22:36 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
2024-03-09 21:22:36 proc_lsst.multi:201 [INFO]  cancelling 16968966 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968966 on provider checkpoint
2024-03-09 21:22:36 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
2024-03-09 21:22:36 proc_lsst.multi:201 [INFO]  cancelling 16968890 on provider astro
proc_lsst.multi INFO: cancelling 16968890 on provider astro
2024-03-09 21:22:36 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
2024-03-09 21:22:36 proc_lsst.multi:201 [INFO]  cancelling 16968904 on provider astro
proc_lsst.multi INFO: cancelling 16968904 on provider astro
2024-03-09 21:22:37 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
2024-03-09 21:22:37 proc_lsst.multi:201 [INFO]  cancelling 16968885 on provider astro
proc_lsst.multi INFO: cancelling 16968885 on provider astro
2024-03-09 21:22:37 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
2024-03-09 21:22:37 proc_lsst.multi:201 [INFO]  cancelling 16968965 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968965 on provider checkpoint
2024-03-09 21:22:37 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
2024-03-09 21:22:37 proc_lsst.multi:201 [INFO]  cancelling 16968907 on provider astro
proc_lsst.multi INFO: cancelling 16968907 on provider astro
2024-03-09 21:22:37 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
2024-03-09 21:22:37 proc_lsst.multi:201 [INFO]  cancelling 16968895 on provider astro
proc_lsst.multi INFO: cancelling 16968895 on provider astro
2024-03-09 21:22:37 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
2024-03-09 21:22:37 proc_lsst.multi:201 [INFO]  cancelling 16968897 on provider astro
proc_lsst.multi INFO: cancelling 16968897 on provider astro
2024-03-09 21:22:37 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
2024-03-09 21:22:37 proc_lsst.multi:201 [INFO]  cancelling 16968929 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968929 on provider escience_a40
2024-03-09 21:22:38 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
2024-03-09 21:22:38 proc_lsst.multi:201 [INFO]  cancelling 16968926 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968926 on provider escience_a40
2024-03-09 21:22:38 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
2024-03-09 21:22:38 proc_lsst.multi:201 [INFO]  cancelling 16968896 on provider astro
proc_lsst.multi INFO: cancelling 16968896 on provider astro
2024-03-09 21:22:38 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
2024-03-09 21:22:38 proc_lsst.multi:201 [INFO]  cancelling 16968978 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968978 on provider checkpoint
2024-03-09 21:22:38 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
2024-03-09 21:22:38 proc_lsst.multi:201 [INFO]  cancelling 16968902 on provider astro
proc_lsst.multi INFO: cancelling 16968902 on provider astro
2024-03-09 21:22:38 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
2024-03-09 21:22:38 proc_lsst.multi:201 [INFO]  cancelling 16968972 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968972 on provider checkpoint
2024-03-09 21:22:39 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
2024-03-09 21:22:39 proc_lsst.multi:201 [INFO]  cancelling 16968891 on provider astro
proc_lsst.multi INFO: cancelling 16968891 on provider astro
2024-03-09 21:22:39 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
2024-03-09 21:22:39 proc_lsst.multi:201 [INFO]  cancelling 16968964 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968964 on provider checkpoint
2024-03-09 21:22:39 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
2024-03-09 21:22:39 proc_lsst.multi:201 [INFO]  cancelling 16968928 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968928 on provider escience_a40
2024-03-09 21:22:39 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
2024-03-09 21:22:39 proc_lsst.multi:201 [INFO]  cancelling 16968941 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968941 on provider escience_a40
2024-03-09 21:22:39 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
2024-03-09 21:22:39 proc_lsst.multi:201 [INFO]  cancelling 16968905 on provider astro
proc_lsst.multi INFO: cancelling 16968905 on provider astro
2024-03-09 21:22:39 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
2024-03-09 21:22:39 proc_lsst.multi:201 [INFO]  cancelling 16968920 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16968920 on provider escience_rtx6k
2024-03-09 21:22:40 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
2024-03-09 21:22:40 proc_lsst.multi:201 [INFO]  cancelling 16968922 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16968922 on provider escience_rtx6k
2024-03-09 21:22:40 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
2024-03-09 21:22:40 proc_lsst.multi:201 [INFO]  cancelling 16968943 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968943 on provider escience_a40
2024-03-09 21:22:40 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
2024-03-09 21:22:40 proc_lsst.multi:201 [INFO]  cancelling 16968936 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968936 on provider escience_a40
2024-03-09 21:22:40 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
2024-03-09 21:22:40 proc_lsst.multi:201 [INFO]  cancelling 16968892 on provider astro
proc_lsst.multi INFO: cancelling 16968892 on provider astro
2024-03-09 21:22:40 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
2024-03-09 21:22:40 proc_lsst.multi:201 [INFO]  cancelling 16968938 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968938 on provider escience_a40
2024-03-09 21:22:40 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
2024-03-09 21:22:40 proc_lsst.multi:201 [INFO]  cancelling 16968971 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968971 on provider checkpoint
2024-03-09 21:22:41 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
2024-03-09 21:22:41 proc_lsst.multi:201 [INFO]  cancelling 16968937 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968937 on provider escience_a40
2024-03-09 21:22:41 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
2024-03-09 21:22:41 proc_lsst.multi:201 [INFO]  cancelling 16968948 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968948 on provider checkpoint
2024-03-09 21:22:41 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
2024-03-09 21:22:41 proc_lsst.multi:201 [INFO]  cancelling 16968940 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968940 on provider escience_a40
2024-03-09 21:22:41 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
2024-03-09 21:22:41 proc_lsst.multi:201 [INFO]  cancelling 16968961 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968961 on provider checkpoint
2024-03-09 21:22:41 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
2024-03-09 21:22:41 proc_lsst.multi:201 [INFO]  cancelling 16968917 on provider astro
proc_lsst.multi INFO: cancelling 16968917 on provider astro
2024-03-09 21:22:42 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
2024-03-09 21:22:42 proc_lsst.multi:201 [INFO]  cancelling 16968899 on provider astro
proc_lsst.multi INFO: cancelling 16968899 on provider astro
2024-03-09 21:22:42 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
2024-03-09 21:22:42 proc_lsst.multi:201 [INFO]  cancelling 16968893 on provider astro
proc_lsst.multi INFO: cancelling 16968893 on provider astro
2024-03-09 21:22:42 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
2024-03-09 21:22:42 proc_lsst.multi:201 [INFO]  cancelling 16968935 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968935 on provider escience_a40
2024-03-09 21:22:42 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
2024-03-09 21:22:42 proc_lsst.multi:201 [INFO]  cancelling 16968960 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968960 on provider checkpoint
2024-03-09 21:22:42 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
2024-03-09 21:22:42 proc_lsst.multi:201 [INFO]  cancelling 16968925 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968925 on provider escience_a40
2024-03-09 21:22:42 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
2024-03-09 21:22:42 proc_lsst.multi:201 [INFO]  cancelling 16968930 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968930 on provider escience_a40
2024-03-09 21:22:43 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
2024-03-09 21:22:43 proc_lsst.multi:201 [INFO]  cancelling 16968889 on provider astro
proc_lsst.multi INFO: cancelling 16968889 on provider astro
parsl.dataflow.dflow INFO: Shutting down executor multi
2024-03-09 21:22:43 proc_lsst.multi:40 [INFO]  Cancelling all provider resources
proc_lsst.multi INFO: Cancelling all provider resources
2024-03-09 21:22:43 proc_lsst.multi:47 [INFO]  new jobs since last cancel ['16968944', '16968945', '16968946', '16968947', '16968948', '16968949', '16968950', '16968951', '16968952', '16968953', '16968954', '16968955', '16968956', '16968957', '16968958', '16968959', '16968960', '16968961', '16968962', '16968963', '16968964', '16968965', '16968966', '16968967', '16968968', '16968969', '16968970', '16968971', '16968972', '16968973', '16968974', '16968975', '16968976', '16968977', '16968978', '16968979', '16968980', '16968884', '16968885', '16968886', '16968887', '16968888', '16968889', '16968890', '16968891', '16968892', '16968893', '16968894', '16968895', '16968896', '16968897', '16968898', '16968899', '16968900', '16968901', '16968902', '16968903', '16968904', '16968905', '16968906', '16968907', '16968909', '16968911', '16968912', '16968913', '16968917', '16968918', '16968924', '16968925', '16968926', '16968927', '16968928', '16968929', '16968930', '16968931', '16968932', '16968933', '16968934', '16968935', '16968936', '16968937', '16968938', '16968939', '16968940', '16968941', '16968942', '16968943', '16968919', '16968920', '16968921', '16968922', '16968923', '51896']
proc_lsst.multi INFO: new jobs since last cancel ['16968944', '16968945', '16968946', '16968947', '16968948', '16968949', '16968950', '16968951', '16968952', '16968953', '16968954', '16968955', '16968956', '16968957', '16968958', '16968959', '16968960', '16968961', '16968962', '16968963', '16968964', '16968965', '16968966', '16968967', '16968968', '16968969', '16968970', '16968971', '16968972', '16968973', '16968974', '16968975', '16968976', '16968977', '16968978', '16968979', '16968980', '16968884', '16968885', '16968886', '16968887', '16968888', '16968889', '16968890', '16968891', '16968892', '16968893', '16968894', '16968895', '16968896', '16968897', '16968898', '16968899', '16968900', '16968901', '16968902', '16968903', '16968904', '16968905', '16968906', '16968907', '16968909', '16968911', '16968912', '16968913', '16968917', '16968918', '16968924', '16968925', '16968926', '16968927', '16968928', '16968929', '16968930', '16968931', '16968932', '16968933', '16968934', '16968935', '16968936', '16968937', '16968938', '16968939', '16968940', '16968941', '16968942', '16968943', '16968919', '16968920', '16968921', '16968922', '16968923', '51896']
2024-03-09 21:22:43 proc_lsst.multi:146 [INFO]  found job 16968944 in provider checkpoint
proc_lsst.multi INFO: found job 16968944 in provider checkpoint
2024-03-09 21:22:43 proc_lsst.multi:201 [INFO]  cancelling 16968944 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968944 on provider checkpoint
2024-03-09 21:22:43 proc_lsst.multi:146 [INFO]  found job 16968945 in provider checkpoint
proc_lsst.multi INFO: found job 16968945 in provider checkpoint
2024-03-09 21:22:43 proc_lsst.multi:201 [INFO]  cancelling 16968945 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968945 on provider checkpoint
2024-03-09 21:22:43 proc_lsst.multi:146 [INFO]  found job 16968946 in provider checkpoint
proc_lsst.multi INFO: found job 16968946 in provider checkpoint
2024-03-09 21:22:43 proc_lsst.multi:201 [INFO]  cancelling 16968946 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968946 on provider checkpoint
2024-03-09 21:22:43 proc_lsst.multi:146 [INFO]  found job 16968947 in provider checkpoint
proc_lsst.multi INFO: found job 16968947 in provider checkpoint
2024-03-09 21:22:43 proc_lsst.multi:201 [INFO]  cancelling 16968947 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968947 on provider checkpoint
2024-03-09 21:22:44 proc_lsst.multi:146 [INFO]  found job 16968948 in provider checkpoint
proc_lsst.multi INFO: found job 16968948 in provider checkpoint
2024-03-09 21:22:44 proc_lsst.multi:201 [INFO]  cancelling 16968948 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968948 on provider checkpoint
2024-03-09 21:22:44 proc_lsst.multi:146 [INFO]  found job 16968949 in provider checkpoint
proc_lsst.multi INFO: found job 16968949 in provider checkpoint
2024-03-09 21:22:44 proc_lsst.multi:201 [INFO]  cancelling 16968949 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968949 on provider checkpoint
2024-03-09 21:22:44 proc_lsst.multi:146 [INFO]  found job 16968950 in provider checkpoint
proc_lsst.multi INFO: found job 16968950 in provider checkpoint
2024-03-09 21:22:44 proc_lsst.multi:201 [INFO]  cancelling 16968950 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968950 on provider checkpoint
2024-03-09 21:22:44 proc_lsst.multi:146 [INFO]  found job 16968951 in provider checkpoint
proc_lsst.multi INFO: found job 16968951 in provider checkpoint
2024-03-09 21:22:44 proc_lsst.multi:201 [INFO]  cancelling 16968951 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968951 on provider checkpoint
2024-03-09 21:22:44 proc_lsst.multi:146 [INFO]  found job 16968952 in provider checkpoint
proc_lsst.multi INFO: found job 16968952 in provider checkpoint
2024-03-09 21:22:44 proc_lsst.multi:201 [INFO]  cancelling 16968952 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968952 on provider checkpoint
2024-03-09 21:22:44 proc_lsst.multi:146 [INFO]  found job 16968953 in provider checkpoint
proc_lsst.multi INFO: found job 16968953 in provider checkpoint
2024-03-09 21:22:44 proc_lsst.multi:201 [INFO]  cancelling 16968953 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968953 on provider checkpoint
2024-03-09 21:22:45 proc_lsst.multi:146 [INFO]  found job 16968954 in provider checkpoint
proc_lsst.multi INFO: found job 16968954 in provider checkpoint
2024-03-09 21:22:45 proc_lsst.multi:201 [INFO]  cancelling 16968954 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968954 on provider checkpoint
2024-03-09 21:22:45 proc_lsst.multi:146 [INFO]  found job 16968955 in provider checkpoint
proc_lsst.multi INFO: found job 16968955 in provider checkpoint
2024-03-09 21:22:45 proc_lsst.multi:201 [INFO]  cancelling 16968955 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968955 on provider checkpoint
2024-03-09 21:22:45 proc_lsst.multi:146 [INFO]  found job 16968956 in provider checkpoint
proc_lsst.multi INFO: found job 16968956 in provider checkpoint
2024-03-09 21:22:45 proc_lsst.multi:201 [INFO]  cancelling 16968956 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968956 on provider checkpoint
2024-03-09 21:22:45 proc_lsst.multi:146 [INFO]  found job 16968957 in provider checkpoint
proc_lsst.multi INFO: found job 16968957 in provider checkpoint
2024-03-09 21:22:45 proc_lsst.multi:201 [INFO]  cancelling 16968957 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968957 on provider checkpoint
2024-03-09 21:22:45 proc_lsst.multi:146 [INFO]  found job 16968958 in provider checkpoint
proc_lsst.multi INFO: found job 16968958 in provider checkpoint
2024-03-09 21:22:45 proc_lsst.multi:201 [INFO]  cancelling 16968958 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968958 on provider checkpoint
2024-03-09 21:22:45 proc_lsst.multi:146 [INFO]  found job 16968959 in provider checkpoint
proc_lsst.multi INFO: found job 16968959 in provider checkpoint
2024-03-09 21:22:46 proc_lsst.multi:201 [INFO]  cancelling 16968959 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968959 on provider checkpoint
2024-03-09 21:22:46 proc_lsst.multi:146 [INFO]  found job 16968960 in provider checkpoint
proc_lsst.multi INFO: found job 16968960 in provider checkpoint
2024-03-09 21:22:46 proc_lsst.multi:201 [INFO]  cancelling 16968960 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968960 on provider checkpoint
2024-03-09 21:22:46 proc_lsst.multi:146 [INFO]  found job 16968961 in provider checkpoint
proc_lsst.multi INFO: found job 16968961 in provider checkpoint
2024-03-09 21:22:46 proc_lsst.multi:201 [INFO]  cancelling 16968961 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968961 on provider checkpoint
2024-03-09 21:22:46 proc_lsst.multi:146 [INFO]  found job 16968962 in provider checkpoint
proc_lsst.multi INFO: found job 16968962 in provider checkpoint
2024-03-09 21:22:46 proc_lsst.multi:201 [INFO]  cancelling 16968962 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968962 on provider checkpoint
2024-03-09 21:22:46 proc_lsst.multi:146 [INFO]  found job 16968963 in provider checkpoint
proc_lsst.multi INFO: found job 16968963 in provider checkpoint
2024-03-09 21:22:46 proc_lsst.multi:201 [INFO]  cancelling 16968963 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968963 on provider checkpoint
2024-03-09 21:22:46 proc_lsst.multi:146 [INFO]  found job 16968964 in provider checkpoint
proc_lsst.multi INFO: found job 16968964 in provider checkpoint
2024-03-09 21:22:46 proc_lsst.multi:201 [INFO]  cancelling 16968964 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968964 on provider checkpoint
2024-03-09 21:22:47 proc_lsst.multi:146 [INFO]  found job 16968965 in provider checkpoint
proc_lsst.multi INFO: found job 16968965 in provider checkpoint
2024-03-09 21:22:47 proc_lsst.multi:201 [INFO]  cancelling 16968965 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968965 on provider checkpoint
2024-03-09 21:22:47 proc_lsst.multi:146 [INFO]  found job 16968966 in provider checkpoint
proc_lsst.multi INFO: found job 16968966 in provider checkpoint
2024-03-09 21:22:47 proc_lsst.multi:201 [INFO]  cancelling 16968966 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968966 on provider checkpoint
2024-03-09 21:22:47 proc_lsst.multi:146 [INFO]  found job 16968967 in provider checkpoint
proc_lsst.multi INFO: found job 16968967 in provider checkpoint
2024-03-09 21:22:47 proc_lsst.multi:201 [INFO]  cancelling 16968967 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968967 on provider checkpoint
2024-03-09 21:22:47 proc_lsst.multi:146 [INFO]  found job 16968968 in provider checkpoint
proc_lsst.multi INFO: found job 16968968 in provider checkpoint
2024-03-09 21:22:47 proc_lsst.multi:201 [INFO]  cancelling 16968968 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968968 on provider checkpoint
2024-03-09 21:22:47 proc_lsst.multi:146 [INFO]  found job 16968969 in provider checkpoint
proc_lsst.multi INFO: found job 16968969 in provider checkpoint
2024-03-09 21:22:47 proc_lsst.multi:201 [INFO]  cancelling 16968969 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968969 on provider checkpoint
2024-03-09 21:22:47 proc_lsst.multi:146 [INFO]  found job 16968970 in provider checkpoint
proc_lsst.multi INFO: found job 16968970 in provider checkpoint
2024-03-09 21:22:47 proc_lsst.multi:201 [INFO]  cancelling 16968970 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968970 on provider checkpoint
2024-03-09 21:22:48 proc_lsst.multi:146 [INFO]  found job 16968971 in provider checkpoint
proc_lsst.multi INFO: found job 16968971 in provider checkpoint
2024-03-09 21:22:48 proc_lsst.multi:201 [INFO]  cancelling 16968971 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968971 on provider checkpoint
2024-03-09 21:22:48 proc_lsst.multi:146 [INFO]  found job 16968972 in provider checkpoint
proc_lsst.multi INFO: found job 16968972 in provider checkpoint
2024-03-09 21:22:48 proc_lsst.multi:201 [INFO]  cancelling 16968972 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968972 on provider checkpoint
2024-03-09 21:22:48 proc_lsst.multi:146 [INFO]  found job 16968973 in provider checkpoint
proc_lsst.multi INFO: found job 16968973 in provider checkpoint
2024-03-09 21:22:48 proc_lsst.multi:201 [INFO]  cancelling 16968973 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968973 on provider checkpoint
2024-03-09 21:22:48 proc_lsst.multi:146 [INFO]  found job 16968974 in provider checkpoint
proc_lsst.multi INFO: found job 16968974 in provider checkpoint
2024-03-09 21:22:48 proc_lsst.multi:201 [INFO]  cancelling 16968974 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968974 on provider checkpoint
2024-03-09 21:22:48 proc_lsst.multi:146 [INFO]  found job 16968975 in provider checkpoint
proc_lsst.multi INFO: found job 16968975 in provider checkpoint
2024-03-09 21:22:48 proc_lsst.multi:201 [INFO]  cancelling 16968975 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968975 on provider checkpoint
2024-03-09 21:22:48 proc_lsst.multi:146 [INFO]  found job 16968976 in provider checkpoint
proc_lsst.multi INFO: found job 16968976 in provider checkpoint
2024-03-09 21:22:48 proc_lsst.multi:201 [INFO]  cancelling 16968976 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968976 on provider checkpoint
2024-03-09 21:22:49 proc_lsst.multi:146 [INFO]  found job 16968977 in provider checkpoint
proc_lsst.multi INFO: found job 16968977 in provider checkpoint
2024-03-09 21:22:49 proc_lsst.multi:201 [INFO]  cancelling 16968977 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968977 on provider checkpoint
2024-03-09 21:22:49 proc_lsst.multi:146 [INFO]  found job 16968978 in provider checkpoint
proc_lsst.multi INFO: found job 16968978 in provider checkpoint
2024-03-09 21:22:49 proc_lsst.multi:201 [INFO]  cancelling 16968978 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968978 on provider checkpoint
2024-03-09 21:22:49 proc_lsst.multi:146 [INFO]  found job 16968979 in provider checkpoint
proc_lsst.multi INFO: found job 16968979 in provider checkpoint
2024-03-09 21:22:49 proc_lsst.multi:201 [INFO]  cancelling 16968979 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968979 on provider checkpoint
2024-03-09 21:22:49 proc_lsst.multi:146 [INFO]  found job 16968980 in provider checkpoint
proc_lsst.multi INFO: found job 16968980 in provider checkpoint
2024-03-09 21:22:49 proc_lsst.multi:201 [INFO]  cancelling 16968980 on provider checkpoint
proc_lsst.multi INFO: cancelling 16968980 on provider checkpoint
2024-03-09 21:22:49 proc_lsst.multi:146 [INFO]  found job 16968884 in provider astro
proc_lsst.multi INFO: found job 16968884 in provider astro
2024-03-09 21:22:49 proc_lsst.multi:201 [INFO]  cancelling 16968884 on provider astro
proc_lsst.multi INFO: cancelling 16968884 on provider astro
2024-03-09 21:22:49 proc_lsst.multi:146 [INFO]  found job 16968885 in provider astro
proc_lsst.multi INFO: found job 16968885 in provider astro
2024-03-09 21:22:49 proc_lsst.multi:201 [INFO]  cancelling 16968885 on provider astro
proc_lsst.multi INFO: cancelling 16968885 on provider astro
2024-03-09 21:22:50 proc_lsst.multi:146 [INFO]  found job 16968886 in provider astro
proc_lsst.multi INFO: found job 16968886 in provider astro
2024-03-09 21:22:50 proc_lsst.multi:201 [INFO]  cancelling 16968886 on provider astro
proc_lsst.multi INFO: cancelling 16968886 on provider astro
2024-03-09 21:22:50 proc_lsst.multi:146 [INFO]  found job 16968887 in provider astro
proc_lsst.multi INFO: found job 16968887 in provider astro
2024-03-09 21:22:50 proc_lsst.multi:201 [INFO]  cancelling 16968887 on provider astro
proc_lsst.multi INFO: cancelling 16968887 on provider astro
2024-03-09 21:22:50 proc_lsst.multi:146 [INFO]  found job 16968888 in provider astro
proc_lsst.multi INFO: found job 16968888 in provider astro
2024-03-09 21:22:50 proc_lsst.multi:201 [INFO]  cancelling 16968888 on provider astro
proc_lsst.multi INFO: cancelling 16968888 on provider astro
2024-03-09 21:22:50 proc_lsst.multi:146 [INFO]  found job 16968889 in provider astro
proc_lsst.multi INFO: found job 16968889 in provider astro
2024-03-09 21:22:50 proc_lsst.multi:201 [INFO]  cancelling 16968889 on provider astro
proc_lsst.multi INFO: cancelling 16968889 on provider astro
2024-03-09 21:22:50 proc_lsst.multi:146 [INFO]  found job 16968890 in provider astro
proc_lsst.multi INFO: found job 16968890 in provider astro
2024-03-09 21:22:50 proc_lsst.multi:201 [INFO]  cancelling 16968890 on provider astro
proc_lsst.multi INFO: cancelling 16968890 on provider astro
2024-03-09 21:22:50 proc_lsst.multi:146 [INFO]  found job 16968891 in provider astro
proc_lsst.multi INFO: found job 16968891 in provider astro
2024-03-09 21:22:50 proc_lsst.multi:201 [INFO]  cancelling 16968891 on provider astro
proc_lsst.multi INFO: cancelling 16968891 on provider astro
2024-03-09 21:22:51 proc_lsst.multi:146 [INFO]  found job 16968892 in provider astro
proc_lsst.multi INFO: found job 16968892 in provider astro
2024-03-09 21:22:51 proc_lsst.multi:201 [INFO]  cancelling 16968892 on provider astro
proc_lsst.multi INFO: cancelling 16968892 on provider astro
2024-03-09 21:22:51 proc_lsst.multi:146 [INFO]  found job 16968893 in provider astro
proc_lsst.multi INFO: found job 16968893 in provider astro
2024-03-09 21:22:51 proc_lsst.multi:201 [INFO]  cancelling 16968893 on provider astro
proc_lsst.multi INFO: cancelling 16968893 on provider astro
2024-03-09 21:22:51 proc_lsst.multi:146 [INFO]  found job 16968894 in provider astro
proc_lsst.multi INFO: found job 16968894 in provider astro
2024-03-09 21:22:51 proc_lsst.multi:201 [INFO]  cancelling 16968894 on provider astro
proc_lsst.multi INFO: cancelling 16968894 on provider astro
2024-03-09 21:22:51 proc_lsst.multi:146 [INFO]  found job 16968895 in provider astro
proc_lsst.multi INFO: found job 16968895 in provider astro
2024-03-09 21:22:51 proc_lsst.multi:201 [INFO]  cancelling 16968895 on provider astro
proc_lsst.multi INFO: cancelling 16968895 on provider astro
2024-03-09 21:22:51 proc_lsst.multi:146 [INFO]  found job 16968896 in provider astro
proc_lsst.multi INFO: found job 16968896 in provider astro
2024-03-09 21:22:51 proc_lsst.multi:201 [INFO]  cancelling 16968896 on provider astro
proc_lsst.multi INFO: cancelling 16968896 on provider astro
2024-03-09 21:22:51 proc_lsst.multi:146 [INFO]  found job 16968897 in provider astro
proc_lsst.multi INFO: found job 16968897 in provider astro
2024-03-09 21:22:51 proc_lsst.multi:201 [INFO]  cancelling 16968897 on provider astro
proc_lsst.multi INFO: cancelling 16968897 on provider astro
2024-03-09 21:22:52 proc_lsst.multi:146 [INFO]  found job 16968898 in provider astro
proc_lsst.multi INFO: found job 16968898 in provider astro
2024-03-09 21:22:52 proc_lsst.multi:201 [INFO]  cancelling 16968898 on provider astro
proc_lsst.multi INFO: cancelling 16968898 on provider astro
2024-03-09 21:22:52 proc_lsst.multi:146 [INFO]  found job 16968899 in provider astro
proc_lsst.multi INFO: found job 16968899 in provider astro
2024-03-09 21:22:52 proc_lsst.multi:201 [INFO]  cancelling 16968899 on provider astro
proc_lsst.multi INFO: cancelling 16968899 on provider astro
2024-03-09 21:22:52 proc_lsst.multi:146 [INFO]  found job 16968900 in provider astro
proc_lsst.multi INFO: found job 16968900 in provider astro
2024-03-09 21:22:52 proc_lsst.multi:201 [INFO]  cancelling 16968900 on provider astro
proc_lsst.multi INFO: cancelling 16968900 on provider astro
2024-03-09 21:22:52 proc_lsst.multi:146 [INFO]  found job 16968901 in provider astro
proc_lsst.multi INFO: found job 16968901 in provider astro
2024-03-09 21:22:52 proc_lsst.multi:201 [INFO]  cancelling 16968901 on provider astro
proc_lsst.multi INFO: cancelling 16968901 on provider astro
2024-03-09 21:22:52 proc_lsst.multi:146 [INFO]  found job 16968902 in provider astro
proc_lsst.multi INFO: found job 16968902 in provider astro
2024-03-09 21:22:52 proc_lsst.multi:201 [INFO]  cancelling 16968902 on provider astro
proc_lsst.multi INFO: cancelling 16968902 on provider astro
2024-03-09 21:22:52 proc_lsst.multi:146 [INFO]  found job 16968903 in provider astro
proc_lsst.multi INFO: found job 16968903 in provider astro
2024-03-09 21:22:52 proc_lsst.multi:201 [INFO]  cancelling 16968903 on provider astro
proc_lsst.multi INFO: cancelling 16968903 on provider astro
2024-03-09 21:22:53 proc_lsst.multi:146 [INFO]  found job 16968904 in provider astro
proc_lsst.multi INFO: found job 16968904 in provider astro
2024-03-09 21:22:53 proc_lsst.multi:201 [INFO]  cancelling 16968904 on provider astro
proc_lsst.multi INFO: cancelling 16968904 on provider astro
2024-03-09 21:22:53 proc_lsst.multi:146 [INFO]  found job 16968905 in provider astro
proc_lsst.multi INFO: found job 16968905 in provider astro
2024-03-09 21:22:53 proc_lsst.multi:201 [INFO]  cancelling 16968905 on provider astro
proc_lsst.multi INFO: cancelling 16968905 on provider astro
2024-03-09 21:22:53 proc_lsst.multi:146 [INFO]  found job 16968906 in provider astro
proc_lsst.multi INFO: found job 16968906 in provider astro
2024-03-09 21:22:53 proc_lsst.multi:201 [INFO]  cancelling 16968906 on provider astro
proc_lsst.multi INFO: cancelling 16968906 on provider astro
2024-03-09 21:22:53 proc_lsst.multi:146 [INFO]  found job 16968907 in provider astro
proc_lsst.multi INFO: found job 16968907 in provider astro
2024-03-09 21:22:53 proc_lsst.multi:201 [INFO]  cancelling 16968907 on provider astro
proc_lsst.multi INFO: cancelling 16968907 on provider astro
2024-03-09 21:22:53 proc_lsst.multi:146 [INFO]  found job 16968909 in provider astro
proc_lsst.multi INFO: found job 16968909 in provider astro
2024-03-09 21:22:53 proc_lsst.multi:201 [INFO]  cancelling 16968909 on provider astro
proc_lsst.multi INFO: cancelling 16968909 on provider astro
2024-03-09 21:22:53 proc_lsst.multi:146 [INFO]  found job 16968911 in provider astro
proc_lsst.multi INFO: found job 16968911 in provider astro
2024-03-09 21:22:53 proc_lsst.multi:201 [INFO]  cancelling 16968911 on provider astro
proc_lsst.multi INFO: cancelling 16968911 on provider astro
2024-03-09 21:22:54 proc_lsst.multi:146 [INFO]  found job 16968912 in provider astro
proc_lsst.multi INFO: found job 16968912 in provider astro
2024-03-09 21:22:54 proc_lsst.multi:201 [INFO]  cancelling 16968912 on provider astro
proc_lsst.multi INFO: cancelling 16968912 on provider astro
2024-03-09 21:22:54 proc_lsst.multi:146 [INFO]  found job 16968913 in provider astro
proc_lsst.multi INFO: found job 16968913 in provider astro
2024-03-09 21:22:54 proc_lsst.multi:201 [INFO]  cancelling 16968913 on provider astro
proc_lsst.multi INFO: cancelling 16968913 on provider astro
2024-03-09 21:22:54 proc_lsst.multi:146 [INFO]  found job 16968917 in provider astro
proc_lsst.multi INFO: found job 16968917 in provider astro
2024-03-09 21:22:54 proc_lsst.multi:201 [INFO]  cancelling 16968917 on provider astro
proc_lsst.multi INFO: cancelling 16968917 on provider astro
2024-03-09 21:22:54 proc_lsst.multi:146 [INFO]  found job 16968918 in provider astro
proc_lsst.multi INFO: found job 16968918 in provider astro
2024-03-09 21:22:54 proc_lsst.multi:201 [INFO]  cancelling 16968918 on provider astro
proc_lsst.multi INFO: cancelling 16968918 on provider astro
2024-03-09 21:22:54 proc_lsst.multi:146 [INFO]  found job 16968924 in provider escience_a40
proc_lsst.multi INFO: found job 16968924 in provider escience_a40
2024-03-09 21:22:54 proc_lsst.multi:201 [INFO]  cancelling 16968924 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968924 on provider escience_a40
2024-03-09 21:22:55 proc_lsst.multi:146 [INFO]  found job 16968925 in provider escience_a40
proc_lsst.multi INFO: found job 16968925 in provider escience_a40
2024-03-09 21:22:55 proc_lsst.multi:201 [INFO]  cancelling 16968925 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968925 on provider escience_a40
2024-03-09 21:22:57 proc_lsst.multi:146 [INFO]  found job 16968926 in provider escience_a40
proc_lsst.multi INFO: found job 16968926 in provider escience_a40
2024-03-09 21:22:57 proc_lsst.multi:201 [INFO]  cancelling 16968926 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968926 on provider escience_a40
2024-03-09 21:22:57 proc_lsst.multi:146 [INFO]  found job 16968927 in provider escience_a40
proc_lsst.multi INFO: found job 16968927 in provider escience_a40
2024-03-09 21:22:57 proc_lsst.multi:201 [INFO]  cancelling 16968927 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968927 on provider escience_a40
2024-03-09 21:22:57 proc_lsst.multi:146 [INFO]  found job 16968928 in provider escience_a40
proc_lsst.multi INFO: found job 16968928 in provider escience_a40
2024-03-09 21:22:57 proc_lsst.multi:201 [INFO]  cancelling 16968928 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968928 on provider escience_a40
2024-03-09 21:22:57 proc_lsst.multi:146 [INFO]  found job 16968929 in provider escience_a40
proc_lsst.multi INFO: found job 16968929 in provider escience_a40
2024-03-09 21:22:57 proc_lsst.multi:201 [INFO]  cancelling 16968929 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968929 on provider escience_a40
2024-03-09 21:22:58 proc_lsst.multi:146 [INFO]  found job 16968930 in provider escience_a40
proc_lsst.multi INFO: found job 16968930 in provider escience_a40
2024-03-09 21:22:58 proc_lsst.multi:201 [INFO]  cancelling 16968930 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968930 on provider escience_a40
2024-03-09 21:22:58 proc_lsst.multi:146 [INFO]  found job 16968931 in provider escience_a40
proc_lsst.multi INFO: found job 16968931 in provider escience_a40
2024-03-09 21:22:58 proc_lsst.multi:201 [INFO]  cancelling 16968931 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968931 on provider escience_a40
2024-03-09 21:22:58 proc_lsst.multi:146 [INFO]  found job 16968932 in provider escience_a40
proc_lsst.multi INFO: found job 16968932 in provider escience_a40
2024-03-09 21:22:58 proc_lsst.multi:201 [INFO]  cancelling 16968932 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968932 on provider escience_a40
2024-03-09 21:22:58 proc_lsst.multi:146 [INFO]  found job 16968933 in provider escience_a40
proc_lsst.multi INFO: found job 16968933 in provider escience_a40
2024-03-09 21:22:58 proc_lsst.multi:201 [INFO]  cancelling 16968933 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968933 on provider escience_a40
2024-03-09 21:22:58 proc_lsst.multi:146 [INFO]  found job 16968934 in provider escience_a40
proc_lsst.multi INFO: found job 16968934 in provider escience_a40
2024-03-09 21:22:58 proc_lsst.multi:201 [INFO]  cancelling 16968934 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968934 on provider escience_a40
2024-03-09 21:22:58 proc_lsst.multi:146 [INFO]  found job 16968935 in provider escience_a40
proc_lsst.multi INFO: found job 16968935 in provider escience_a40
2024-03-09 21:22:58 proc_lsst.multi:201 [INFO]  cancelling 16968935 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968935 on provider escience_a40
2024-03-09 21:22:59 proc_lsst.multi:146 [INFO]  found job 16968936 in provider escience_a40
proc_lsst.multi INFO: found job 16968936 in provider escience_a40
2024-03-09 21:22:59 proc_lsst.multi:201 [INFO]  cancelling 16968936 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968936 on provider escience_a40
2024-03-09 21:22:59 proc_lsst.multi:146 [INFO]  found job 16968937 in provider escience_a40
proc_lsst.multi INFO: found job 16968937 in provider escience_a40
2024-03-09 21:22:59 proc_lsst.multi:201 [INFO]  cancelling 16968937 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968937 on provider escience_a40
2024-03-09 21:22:59 proc_lsst.multi:146 [INFO]  found job 16968938 in provider escience_a40
proc_lsst.multi INFO: found job 16968938 in provider escience_a40
2024-03-09 21:22:59 proc_lsst.multi:201 [INFO]  cancelling 16968938 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968938 on provider escience_a40
2024-03-09 21:22:59 proc_lsst.multi:146 [INFO]  found job 16968939 in provider escience_a40
proc_lsst.multi INFO: found job 16968939 in provider escience_a40
2024-03-09 21:22:59 proc_lsst.multi:201 [INFO]  cancelling 16968939 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968939 on provider escience_a40
2024-03-09 21:22:59 proc_lsst.multi:146 [INFO]  found job 16968940 in provider escience_a40
proc_lsst.multi INFO: found job 16968940 in provider escience_a40
2024-03-09 21:22:59 proc_lsst.multi:201 [INFO]  cancelling 16968940 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968940 on provider escience_a40
2024-03-09 21:23:00 proc_lsst.multi:146 [INFO]  found job 16968941 in provider escience_a40
proc_lsst.multi INFO: found job 16968941 in provider escience_a40
2024-03-09 21:23:00 proc_lsst.multi:201 [INFO]  cancelling 16968941 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968941 on provider escience_a40
2024-03-09 21:23:00 proc_lsst.multi:146 [INFO]  found job 16968942 in provider escience_a40
proc_lsst.multi INFO: found job 16968942 in provider escience_a40
2024-03-09 21:23:00 proc_lsst.multi:201 [INFO]  cancelling 16968942 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968942 on provider escience_a40
2024-03-09 21:23:00 proc_lsst.multi:146 [INFO]  found job 16968943 in provider escience_a40
proc_lsst.multi INFO: found job 16968943 in provider escience_a40
2024-03-09 21:23:00 proc_lsst.multi:201 [INFO]  cancelling 16968943 on provider escience_a40
proc_lsst.multi INFO: cancelling 16968943 on provider escience_a40
2024-03-09 21:23:00 proc_lsst.multi:146 [INFO]  found job 16968919 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968919 in provider escience_rtx6k
2024-03-09 21:23:00 proc_lsst.multi:201 [INFO]  cancelling 16968919 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16968919 on provider escience_rtx6k
2024-03-09 21:23:00 proc_lsst.multi:146 [INFO]  found job 16968920 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968920 in provider escience_rtx6k
2024-03-09 21:23:00 proc_lsst.multi:201 [INFO]  cancelling 16968920 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16968920 on provider escience_rtx6k
2024-03-09 21:23:00 proc_lsst.multi:146 [INFO]  found job 16968921 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968921 in provider escience_rtx6k
2024-03-09 21:23:00 proc_lsst.multi:201 [INFO]  cancelling 16968921 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16968921 on provider escience_rtx6k
2024-03-09 21:23:01 proc_lsst.multi:146 [INFO]  found job 16968922 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968922 in provider escience_rtx6k
2024-03-09 21:23:01 proc_lsst.multi:201 [INFO]  cancelling 16968922 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16968922 on provider escience_rtx6k
2024-03-09 21:23:01 proc_lsst.multi:146 [INFO]  found job 16968923 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16968923 in provider escience_rtx6k
2024-03-09 21:23:01 proc_lsst.multi:201 [INFO]  cancelling 16968923 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16968923 on provider escience_rtx6k
2024-03-09 21:23:01 proc_lsst.multi:146 [INFO]  found job 51896 in provider local
proc_lsst.multi INFO: found job 51896 in provider local
2024-03-09 21:23:01 proc_lsst.multi:201 [INFO]  cancelling 51896 on provider local
proc_lsst.multi INFO: cancelling 51896 on provider local
parsl.providers.local.local DEBUG: Terminating job/proc_id: 51896
parsl.providers.local.local WARNING: Failed to kill PID: 51896 and child processes on local
2024-03-09 21:23:02 proc_lsst.multi:50 [INFO]  no new jobs since last cancel, resuming executor shutdown
proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown
parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown
parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt
parsl.dataflow.dflow INFO: Shut down executor multi
parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal
parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete
parsl.executors.threads DEBUG: Done with executor shutdown
parsl.dataflow.dflow INFO: Shut down executor _parsl_internal
parsl.dataflow.dflow INFO: Terminated executors
parsl.dataflow.dflow INFO: DFK cleanup complete
parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread
lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 1752.0025 seconds
lsst.ctrl.bps.drivers INFO: Run 'DEEP_20190507_A1c_science#step1_20240309T205315Z' submitted for execution with id 'None'
lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 1752.0147 seconds; current memory usage: 0.396 Gibyte, delta: 0.009 Gibyte, peak delta: 0.009 Gibyte
lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 1782.9337 seconds; current memory usage: 0.396 Gibyte, delta: 0.256 Gibyte, peak delta: 0.256 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.396 Gibyte (main), 0.396 Gibyte (largest child process)
Run Id: None
Run Name: DEEP_20190507_A1c_science#step1_20240309T205315Z
parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up

real	29m46.449s
user	7m49.021s
sys	5m10.306s